Emotion Recognition in Conversations Using Multimodality Approach

Abstract

" Emotions play an important role in communication and people interactions. Even in face- to-face conversations, it's challenging to accurately interpret the feelings of the other person due to the complexity of emotions recognition by analysing the person’s behavior. These emotions involve various signals like tone of the voice, gestures and can vary between individuals. When it comes to recognizing emotions in digital conversations, the difficulty increases more.

In response to those challenges, this study proposes a novel approach based upon the integration of three modalities - text, audio, and visual cues - that generated during virtual conversations. Through experimentation & evaluation, the research demonstrates that the accuracy achieved in emotion prediction using of any two or even a single modality alone is surpassed by the some approaches done by integrating of text, audio, and visual data. The proposed Ensemble Technique in the research shows a 26% accuracy improvement compared to the least accuracy unimodality model used in the experiment. During the evaluation process, input was gathered from both academic professionals and industry experts and conducted a thorough self evaluation of the research. The aim of considering all these quantitative and qualitative measures was is to recognize strengths and identify areas for future enhancement."