A method for lexical tone classification in audio-visual speech

• Autor(a) principal: João Vítor Possamai de Menezes
• Data de Publicação: 2020
• Outros Autores: Maria Mendes Cantoni, Denis Burnham, Adriano Vilela Barbosa
• Tipo de documento: Artigo
• Idioma: eng
• Título da fonte: Repositório Institucional da UFMG
• Texto Completo: https://doi.org/10.20396/joss.v9i00.14960; http://hdl.handle.net/1843/49361; http://orcid.org/0000-0002-7612-9754; https://orcid.org/0000-0001-9515-1802; http://orcid.org/0000-0002-1980-3458; http://orcid.org/0000-0003-1083-8256
• topic: Multimodal speech; Lexical tone; Cantonese language; Statistical learning; Linear discriminant analysis; Fala

This work presents a method for lexical tone classification in audio-visual speech. The method is applied to a speech data set consisting of syllables and words produced by a female native speaker of Cantonese. The data were recorded in an audio-visual speech production experiment. The visual component of speech was measured by tracking the positions of active markers placed on the speaker's face, whereas the acoustic component was measured with an ordinary microphone. A pitch tracking algorithm is used to estimate F0 from the acoustic signal. A procedure for head motion compensation is applied to the tracked marker positions in order to separate the head and face motion components. The data are then organized into four signal groups: F0, Face, Head, Face+Head. The signals in each of these groups are parameterized by means of a polynomial approximation and then used to train an LDA (Linear Discriminant Analysis) classifier that maps the input signals into one of the output classes (the lexical tones of the language). One classifier is trained for each signal group. The ability of each signal group to predict the correct lexical tones was assessed by the accuracy of the corresponding LDA classifier. The accuracy of the classifiers was obtained by means of a k-fold cross validation method. The classifiers for all signal groups performed above chance, with F0 achieving the highest accuracy, followed by Face+Head, Face, and Head, respectively. The differences in performance between all signal groups were statistically significant.