Empowering deaf-hearing communication: exploring synergies between predictive and generative AI-based strategies towards (Portuguese) Sign Language interpretation
Autor(a) principal: | |
---|---|
Data de Publicação: | 2023 |
Outros Autores: | , , , , , , , , |
Tipo de documento: | Artigo |
Idioma: | eng |
Título da fonte: | Repositório Científico de Acesso Aberto de Portugal (Repositórios Cientìficos) |
Texto Completo: | http://hdl.handle.net/10400.26/47818 |
Resumo: | Communication between Deaf and hearing individuals remains a persistent challenge requiring attention to foster inclusivity. Despite notable efforts in the development of digital solutions for sign language recognition (SLR), several issues persist, such as cross-platform interoperability and strategies for tokenizing signs to enable continuous conversations and coherent sentence construction. To address such issues, this paper proposes a non-invasive Portuguese Sign Language (Língua Gestual Portuguesa or LGP) interpretation system-as-a-service, leveraging skeletal posture sequence inference powered by long-short term memory (LSTM) architectures. To address the scarcity of examples during machine learning (ML) model training, dataset augmentation strategies are explored. Additionally, a buffer-based interaction technique is introduced to facilitate LGP terms tokenization. This technique provides real-time feedback to users, allowing them to gauge the time remaining to complete a sign, which aids in the construction of grammatically coherent sentences based on inferred terms/words. To support human-like conditioning rules for interpretation, a large language model (LLM) service is integrated. Experiments reveal that LSTM-based neural networks, trained with 50 LGP terms and subjected to data augmentation, achieved accuracy levels ranging from 80% to 95.6%. Users unanimously reported a high level of intuition when using the buffer-based interaction strategy for terms/words tokenization. Furthermore, tests with an LLM—specifically ChatGPT—demonstrated promising semantic correlation rates in generated sentences, comparable to expected sentences. |
id |
RCAP_78e10c670c8bac8adf39d8036fd472a3 |
---|---|
oai_identifier_str |
oai:comum.rcaap.pt:10400.26/47818 |
network_acronym_str |
RCAP |
network_name_str |
Repositório Científico de Acesso Aberto de Portugal (Repositórios Cientìficos) |
repository_id_str |
7160 |
spelling |
Empowering deaf-hearing communication: exploring synergies between predictive and generative AI-based strategies towards (Portuguese) Sign Language interpretationCommunication between Deaf and hearing individuals remains a persistent challenge requiring attention to foster inclusivity. Despite notable efforts in the development of digital solutions for sign language recognition (SLR), several issues persist, such as cross-platform interoperability and strategies for tokenizing signs to enable continuous conversations and coherent sentence construction. To address such issues, this paper proposes a non-invasive Portuguese Sign Language (Língua Gestual Portuguesa or LGP) interpretation system-as-a-service, leveraging skeletal posture sequence inference powered by long-short term memory (LSTM) architectures. To address the scarcity of examples during machine learning (ML) model training, dataset augmentation strategies are explored. Additionally, a buffer-based interaction technique is introduced to facilitate LGP terms tokenization. This technique provides real-time feedback to users, allowing them to gauge the time remaining to complete a sign, which aids in the construction of grammatically coherent sentences based on inferred terms/words. To support human-like conditioning rules for interpretation, a large language model (LLM) service is integrated. Experiments reveal that LSTM-based neural networks, trained with 50 LGP terms and subjected to data augmentation, achieved accuracy levels ranging from 80% to 95.6%. Users unanimously reported a high level of intuition when using the buffer-based interaction strategy for terms/words tokenization. Furthermore, tests with an LLM—specifically ChatGPT—demonstrated promising semantic correlation rates in generated sentences, comparable to expected sentences.Repositório ComumAdão, TelmoOliveira, JoãoShahrabadi, SomayehJesus, HugoFernandes, MarcoCosta, ÂngeloFerreira, VâniaGonçalves, Martinho FradeiraGuevara Lopez, Miguel AngelPeres, Emanuel2023-11-03T16:10:23Z20232023-01-01T00:00:00Zinfo:eu-repo/semantics/publishedVersioninfo:eu-repo/semantics/articleapplication/pdfhttp://hdl.handle.net/10400.26/47818engAdão, T., Oliveira, J., Shahrabadi, S., Jesus, H., Fernandes, M., Costa, Â., Ferreira, V., et al. (2023). Empowering Deaf-Hearing Communication: Exploring Synergies between Predictive and Generative AI-Based Strategies towards (Portuguese) Sign Language Interpretation. Journal of Imaging, 9(11), 235. https://doi.org/10.3390/jimaging91102352313-433Xhttps://doi.org/10.3390/jimaging9110235info:eu-repo/semantics/openAccessreponame:Repositório Científico de Acesso Aberto de Portugal (Repositórios Cientìficos)instname:Agência para a Sociedade do Conhecimento (UMIC) - FCT - Sociedade da Informaçãoinstacron:RCAAP2023-11-21T09:57:56Zoai:comum.rcaap.pt:10400.26/47818Portal AgregadorONGhttps://www.rcaap.pt/oai/openaireopendoar:71602024-03-19T23:13:18.832637Repositório Científico de Acesso Aberto de Portugal (Repositórios Cientìficos) - Agência para a Sociedade do Conhecimento (UMIC) - FCT - Sociedade da Informaçãofalse |
dc.title.none.fl_str_mv |
Empowering deaf-hearing communication: exploring synergies between predictive and generative AI-based strategies towards (Portuguese) Sign Language interpretation |
title |
Empowering deaf-hearing communication: exploring synergies between predictive and generative AI-based strategies towards (Portuguese) Sign Language interpretation |
spellingShingle |
Empowering deaf-hearing communication: exploring synergies between predictive and generative AI-based strategies towards (Portuguese) Sign Language interpretation Adão, Telmo |
title_short |
Empowering deaf-hearing communication: exploring synergies between predictive and generative AI-based strategies towards (Portuguese) Sign Language interpretation |
title_full |
Empowering deaf-hearing communication: exploring synergies between predictive and generative AI-based strategies towards (Portuguese) Sign Language interpretation |
title_fullStr |
Empowering deaf-hearing communication: exploring synergies between predictive and generative AI-based strategies towards (Portuguese) Sign Language interpretation |
title_full_unstemmed |
Empowering deaf-hearing communication: exploring synergies between predictive and generative AI-based strategies towards (Portuguese) Sign Language interpretation |
title_sort |
Empowering deaf-hearing communication: exploring synergies between predictive and generative AI-based strategies towards (Portuguese) Sign Language interpretation |
author |
Adão, Telmo |
author_facet |
Adão, Telmo Oliveira, João Shahrabadi, Somayeh Jesus, Hugo Fernandes, Marco Costa, Ângelo Ferreira, Vânia Gonçalves, Martinho Fradeira Guevara Lopez, Miguel Angel Peres, Emanuel |
author_role |
author |
author2 |
Oliveira, João Shahrabadi, Somayeh Jesus, Hugo Fernandes, Marco Costa, Ângelo Ferreira, Vânia Gonçalves, Martinho Fradeira Guevara Lopez, Miguel Angel Peres, Emanuel |
author2_role |
author author author author author author author author author |
dc.contributor.none.fl_str_mv |
Repositório Comum |
dc.contributor.author.fl_str_mv |
Adão, Telmo Oliveira, João Shahrabadi, Somayeh Jesus, Hugo Fernandes, Marco Costa, Ângelo Ferreira, Vânia Gonçalves, Martinho Fradeira Guevara Lopez, Miguel Angel Peres, Emanuel |
description |
Communication between Deaf and hearing individuals remains a persistent challenge requiring attention to foster inclusivity. Despite notable efforts in the development of digital solutions for sign language recognition (SLR), several issues persist, such as cross-platform interoperability and strategies for tokenizing signs to enable continuous conversations and coherent sentence construction. To address such issues, this paper proposes a non-invasive Portuguese Sign Language (Língua Gestual Portuguesa or LGP) interpretation system-as-a-service, leveraging skeletal posture sequence inference powered by long-short term memory (LSTM) architectures. To address the scarcity of examples during machine learning (ML) model training, dataset augmentation strategies are explored. Additionally, a buffer-based interaction technique is introduced to facilitate LGP terms tokenization. This technique provides real-time feedback to users, allowing them to gauge the time remaining to complete a sign, which aids in the construction of grammatically coherent sentences based on inferred terms/words. To support human-like conditioning rules for interpretation, a large language model (LLM) service is integrated. Experiments reveal that LSTM-based neural networks, trained with 50 LGP terms and subjected to data augmentation, achieved accuracy levels ranging from 80% to 95.6%. Users unanimously reported a high level of intuition when using the buffer-based interaction strategy for terms/words tokenization. Furthermore, tests with an LLM—specifically ChatGPT—demonstrated promising semantic correlation rates in generated sentences, comparable to expected sentences. |
publishDate |
2023 |
dc.date.none.fl_str_mv |
2023-11-03T16:10:23Z 2023 2023-01-01T00:00:00Z |
dc.type.status.fl_str_mv |
info:eu-repo/semantics/publishedVersion |
dc.type.driver.fl_str_mv |
info:eu-repo/semantics/article |
format |
article |
status_str |
publishedVersion |
dc.identifier.uri.fl_str_mv |
http://hdl.handle.net/10400.26/47818 |
url |
http://hdl.handle.net/10400.26/47818 |
dc.language.iso.fl_str_mv |
eng |
language |
eng |
dc.relation.none.fl_str_mv |
Adão, T., Oliveira, J., Shahrabadi, S., Jesus, H., Fernandes, M., Costa, Â., Ferreira, V., et al. (2023). Empowering Deaf-Hearing Communication: Exploring Synergies between Predictive and Generative AI-Based Strategies towards (Portuguese) Sign Language Interpretation. Journal of Imaging, 9(11), 235. https://doi.org/10.3390/jimaging9110235 2313-433X https://doi.org/10.3390/jimaging9110235 |
dc.rights.driver.fl_str_mv |
info:eu-repo/semantics/openAccess |
eu_rights_str_mv |
openAccess |
dc.format.none.fl_str_mv |
application/pdf |
dc.source.none.fl_str_mv |
reponame:Repositório Científico de Acesso Aberto de Portugal (Repositórios Cientìficos) instname:Agência para a Sociedade do Conhecimento (UMIC) - FCT - Sociedade da Informação instacron:RCAAP |
instname_str |
Agência para a Sociedade do Conhecimento (UMIC) - FCT - Sociedade da Informação |
instacron_str |
RCAAP |
institution |
RCAAP |
reponame_str |
Repositório Científico de Acesso Aberto de Portugal (Repositórios Cientìficos) |
collection |
Repositório Científico de Acesso Aberto de Portugal (Repositórios Cientìficos) |
repository.name.fl_str_mv |
Repositório Científico de Acesso Aberto de Portugal (Repositórios Cientìficos) - Agência para a Sociedade do Conhecimento (UMIC) - FCT - Sociedade da Informação |
repository.mail.fl_str_mv |
|
_version_ |
1799135408394600448 |