Multimodal Emotion Recognition in Polish (Student Consortium)

Abstract

Multimodal emotion recognition is a challenging task because emotions can be expressed through various forms and modalities. It can be applied in various fields, for example, human-computer interaction, crime, healthcare, multimedia retrieval, etc. In recent times, neural networks have achieved overwhelming success in determining emotional states. Motivated by these advancements, we present a multimodal emotion recognition system that uses the following modalities: facial expression, speech, and body language. This paper specifies techniques used for the Emotion Recognition on Polish problem based Polish Emotion dataset. To recognise the current state of emotion in the various given videos, preprocessing of data and extraction of robust features is employed. For the given task, we have made use of facial landmark detection, and Mel-frequency cepstral coefficients (MFCCs) for speech audio. The presented data was in the format of variable length videos, subsequently, which led to underperformance by traditional algorithms for classification. Thus, they could not be used. Therefore, we implemented several long short-term memory (LSTM) networks. Each particular modality was trained for its specific LSTM model, in order to return the emotion having the highest probability at a given instance. Finally, each individual model is fused together using a weight average approach, where in context to all the modalities, the emotion having highest probability is the desirable emotion. © 2020 IEEE.