# cross-modal fusion techniques for utterance-level emotion recognition from text and speech

arXiv: https://arxiv.org/abs/2302.02447 · HTML: https://mlsanity.com/paper/2302.02447/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Feb 2023.

## Built on by (most cited first, 5 of 5)

- [SSLCL: An Efficient Model-Agnostic Supervised Contrastive Learning Framework for Emotion Recognition in Conversations](https://mlsanity.com/paper/2310.16676/) (2023)
- [Bimodal Connection Attention Fusion for Speech Emotion Recognition](https://mlsanity.com/paper/2503.05858/) (2025)
- [Heterogeneous bimodal attention fusion for speech emotion recognition](https://mlsanity.com/paper/2503.06405/) (2025)
- [On the Contribution of Lexical Features to Speech Emotion Recognition](https://mlsanity.com/paper/2509.05634/) (2025)
- [RAFM-SER++: A Lightweight Multimodal Emotion Recognition Framework for Real-Time Behavioral Monitoring in Surveillance Systems](https://mlsanity.com/paper/2609.07409/) (2026)

## Abstract (arXiv)

Multimodal emotion recognition (MER) is a fundamental complex research problem due to the uncertainty of human emotional expression and the heterogeneity gap between different modalities. Audio and text modalities are particularly important for a human participant in understanding emotions. Although many successful attempts have been designed multimodal representations for MER, there still exist multiple challenges to be addressed: 1) bridging the heterogeneity gap between multimodal features and model inter- and intra-modal interactions of multiple modalities; 2) effectively and efficiently modelling the contextual dynamics in the conversation sequence. In this paper, we propose Cross-Modal RoBERTa (CM-RoBERTa) model for emotion detection from spoken audio and corresponding transcripts. As the core unit of the CM-RoBERTa, parallel self- and cross- attention is designed to dynamically capture inter- and intra-modal interactions of audio and text. Specially, the mid-level fusion and residual module are employed to model long-term contextual dependencies and learn modality-specific patterns. We evaluate the approach on the MELD dataset and the experimental results show the proposed approach achieves the state-of-art performance on the dataset.