Search
Abstract
Emotion recognition is a crucial aspect of human communication, with applications in fields such as psychology, education, and healthcare. Identifying emotions accurately is challenging, as people use a variety of signals to express and perceive emotions. In this study, we address the problem of multimodal emotion recognition using both audio and video signals, to develop a robust and reliable system that can recognize emotions even when one modality is absent. To achieve this goal, we propose a novel architecture based on well-designed feature extractors for each modality and use model-level fusion based on a TFusion block to combine the information from both sources. To be more efficient in real-world scenarios, we trained our model on a compound dataset consisting of RAVDESS, RML, and eNTERFACE'05. It is then evaluated and compared to the state-of-the-art models. We find that our approach performs close to the modern solutions and can recognize emotions accurately when one of the modalities is missing. Additionally, we have developed a real-time emotion recognition application as a part of this work.
Citations
-
1
CrossRef
-
0
Web of Science
-
1
Scopus
Cite as
Full text
- Publication version
- Accepted or Published Version
- DOI:
- Digital Object Identifier (open in new tab) 10.1016/j.procs.2023.10.247
- License
-
open in new tab
Keywords
Details
- Category:
- Conference activity
- Type:
- publikacja w wydawnictwie zbiorowym recenzowanym (także w materiałach konferencyjnych)
- Language:
- English
- Publication year:
- 2023
- Bibliographic description:
- Woźniak M., Sakowicz M., Ledwosiński K., Rzepkowski J., Czapla P., Zaporowski S.: Bimodal Emotion Recognition Based on Vocal and Facial Features// / : , 2023,
- DOI:
- Digital Object Identifier (open in new tab) 10.1016/j.procs.2023.10.247
- Sources of funding:
-
- Project -
- Verified by:
- Gdańsk University of Technology
seen 64 times