Multimodal emotion evaluation: a physiological model for cost-effective emotion classification

• Main Author: Pinto, Gisela
• Publication Date: 2020
• Other Authors: Carvalho, João M., Barros, Filipa, Soares, Sandra C., Pinho, Armando J., Brás, Susana
• Format: Article
• Language: eng
• Source: Repositório Científico de Acesso Aberto de Portugal (Repositórios Cientìficos)
• Download full: http://hdl.handle.net/10773/28812
• topic: Affective computing; Multimodal; Feature extraction; Random forest; Neural network

Emotional responses are associated with distinct body alterations and are crucial to foster adaptive responses, well-being, and survival. Emotion identification may improve peoples' emotion regulation strategies and interaction with multiple life contexts. Several studies have investigated emotion classification systems, but most of them are based on the analysis of only one, a few, or isolated physiological signals. Understanding how informative the individual signals are and how their combination works would allow to develop more cost-effective, informative, and objective systems for emotion detection, processing, and interpretation. In the present work, electrocardiogram, electromyogram, and electrodermal activity were processed in order to find a physiological model of emotions. Both a unimodal and a multimodal approach were used to analyze what signal, or combination of signals, may better describe an emotional response, using a sample of 55 healthy subjects. The method was divided in: (1) signal preprocessing; (2) feature extraction; (3) classification using random forest and neural networks. Results suggest that the electrocardiogram (ECG) signal is the most effective for emotion classification. Yet, the combination of all signals provides the best emotion identification performance, with all signals providing crucial information for the system. This physiological model of emotions has important research and clinical implications, by providing valuable information about the value and weight of physiological signals for emotional classification, which can critically drive effective evaluation, monitoring and intervention, regarding emotional processing and regulation, considering multiple contexts.