# EmoBed: Strengthening Monomodal Emotion Recognition via Training with Crossmodal Emotion Embeddings

arXiv: https://arxiv.org/abs/1907.10428 · HTML: https://mlsanity.com/paper/1907.10428/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Jul 2019.

## Built on by (most cited first, 8 of 8)

- [Refashioning Emotion Recognition Modelling: The Advent of Generalised Large Models](https://mlsanity.com/paper/2308.11578/) (2023)
- [Gaze-enhanced Crossmodal Embeddings for Emotion Recognition](https://mlsanity.com/paper/2205.00129/) (2022)
- [Exploring Attention Mechanisms for Multimodal Emotion Recognition in an Emergency Call Center Corpus](https://mlsanity.com/paper/2306.07115/) (2023)
- [Toward Affective XAI: Facial Affect Analysis for Understanding Explainable Human-AI Interactions](https://mlsanity.com/paper/2106.08761/) (2021)
- [LineConGraphs: Line Conversation Graphs for Effective Emotion Recognition using Graph Neural Networks](https://mlsanity.com/paper/2312.03756/) (2023)
- [CLHOP: Combined Audio-Video Learning for Horse 3D Pose and Shape Estimation](https://mlsanity.com/paper/2407.01244/) (2024)
- [Emotion Embeddings unicodex2014 Learning Stable and Homogeneous Abstractions from Heterogeneous Affective Datasets](https://mlsanity.com/paper/2308.07871/) (2023)
- [Enhancing Apparent Personality Trait Analysis with Cross-Modal Embeddings](https://mlsanity.com/paper/2405.03846/) (2024)

## Abstract (arXiv)

Despite remarkable advances in emotion recognition, they are severely restrained from either the essentially limited property of the employed single modality, or the synchronous presence of all involved multiple modalities. Motivated by this, we propose a novel crossmodal emotion embedding framework called EmoBed, which aims to leverage the knowledge from other auxiliary modalities to improve the performance of an emotion recognition system at hand. The framework generally includes two main learning components, i. e., joint multimodal training and crossmodal training. Both of them tend to explore the underlying semantic emotion information but with a shared recognition network or with a shared emotion embedding space, respectively. In doing this, the enhanced system trained with this approach can efficiently make use of the complementary information from other modalities. Nevertheless, the presence of these auxiliary modalities is not demanded during inference. To empirically investigate the effectiveness and robustness of the proposed framework, we perform extensive experiments on the two benchmark databases RECOLA and OMG-Emotion for the tasks of dimensional emotion regression and categorical emotion classification, respectively. The obtained results show that the proposed framework significantly outperforms related baselines in monomodal inference, and are also competitive or superior to the recently reported systems, which emphasises the importance of the proposed crossmodal learning for emotion recognition.