Empowering deaf-hearing communication: exploring synergies between predictive and generative ai-based strategies towards (portuguese) sign language interpretation

• Autor(a) principal: Adão, Telmo
• Data de Publicação: 2011
• Outros Autores: Oliveira, Joao, Shahrabadi, Somayeh, Jesus, Hugo, Fernandes, Marco Paulo Sampaio, Costa, Angelo, Gonçalves, Martinho Fradeira, Lopez, Miguel A.G., Peres, Emanuel, Magalhães, Luis G.
• Tipo de documento: Artigo
• Idioma: eng
• Título da fonte: Repositório Científico de Acesso Aberto de Portugal (Repositórios Cientìficos)
• Texto Completo: http://hdl.handle.net/10198/4957
• topic: Sign language recognition (SLR); Portuguese sign language; Video-based motion analytics; Machine learning (ML); Long-short term memory (LSTM); Large language models (LLM); Generative pre-trained transformer (GPT); Deaf-hearing communication; Inclusion

Communication between Deaf and hearing individuals remains a persistent challenge requiring attention to foster inclusivity. Despite notable efforts in the development of digital solutions for sign language recognition (SLR), several issues persist, such as cross-platform interoperability and strategies for tokenizing signs to enable continuous conversations and coherent sentence construction. To address such issues, this paper proposes a non-invasive Portuguese Sign Language (Língua Gestual Portuguesa or LGP) interpretation system-as-a-service, leveraging skeletal posture sequence inference powered by long-short term memory (LSTM) architectures. To address the scarcity of examples during machine learning (ML) model training, dataset augmentation strategies are explored. Additionally, a buffer-based interaction technique is introduced to facilitate LGP terms tokenization. This technique provides real-time feedback to users, allowing them to gauge the time remaining to complete a sign, which aids in the construction of grammatically coherent sentences based on inferred terms/words. To support human-like conditioning rules for interpretation, a large language model (LLM) service is integrated. Experiments reveal that LSTM-based neural networks, trained with 50 LGP terms and subjected to data augmentation, achieved accuracy levels ranging from 80% to 95.6%. Users unanimously reported a high level of intuition when using the buffer-based interaction strategy for terms/words tokenization. Furthermore, tests with an LLM—specifically ChatGPT—demonstrated promising semantic correlation rates in generated sentences, comparable to expected sentences.