# Enhancing Multimodal Emotion Recognition through Multi-Granularity Cross-Modal Alignment

arXiv: https://arxiv.org/abs/2412.20821 · HTML: https://mlsanity.com/paper/2412.20821/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 26 references that are in the map.
- First posted on arXiv 30 Dec 2024.

## Builds on (most cited first, 14 of 14)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [MAP: Multimodal Uncertainty-Aware Vision-Language Pre-training Model](https://mlsanity.com/paper/2210.05335/) (2022)
- [Group Gated Fusion on Attention-based Bidirectional Alignment for Multimodal Emotion Recognition](https://mlsanity.com/paper/2201.06309/) (2022)
- [Efficient Speech Emotion Recognition Using Multi-Scale CNN and Attention](https://mlsanity.com/paper/2106.04133/) (2021)
- [Touch Your Heart: A Tone-aware Chatbot for Customer Care on Social Media](https://mlsanity.com/paper/1803.02952/) (2018)
- [Learning Fine-Grained Cross Modality Excitement for Speech Emotion Recognition](https://mlsanity.com/paper/2010.12733/) (2020)
- [Fine-grained Disentangled Representation Learning for Multimodal Emotion Recognition](https://mlsanity.com/paper/2312.13567/) (2023)
- [Using Auxiliary Tasks In Multimodal Fusion Of Wav2vec 2.0 And BERT For Multimodal Emotion Recognition](https://mlsanity.com/paper/2302.13661/) (2023)
- [Knowledge-aware Bayesian Co-attention for Multimodal Emotion Recognition](https://mlsanity.com/paper/2302.09856/) (2023)
- [Multi-Classifier Interactive Learning for Ambiguous Speech Emotion Recognition](https://mlsanity.com/paper/2012.05429/) (2020)
- [Convolutional Attention Networks for Multimodal Emotion Recognition from Speech and Text Data](https://mlsanity.com/paper/1805.06606/) (2018)
- [Supervised Contrastive Learning with Nearest Neighbor Search for Speech Emotion Recognition](https://mlsanity.com/paper/2308.16485/) (2023)

## Built on by (most cited first, 5 of 5)

- [OmniVox: Zero-Shot Emotion Recognition with Omni-LLMs](https://mlsanity.com/paper/2503.21480/) (2025)
- [EmoEUS: Uncertainty Supervision for Multimodal Emotion Recognition in Conversation](https://mlsanity.com/paper/2607.18336/) (2026)
- [Learning to Prefer Reliably: Error-Augmented Emotion Preference Optimization with Calibrated Fusion](https://mlsanity.com/paper/2608.24730/) (2026)
- [Quality-Controlled Multimodal Emotion Recognition in Conversations with Identity-Based Transfer Learning and MAMBA Fusion](https://mlsanity.com/paper/2511.14969/) (2025)
- [BiCFlow-MER: Orchestrating Discriminative and Generative Multimodal Emotion Recognition via Conditional Transport](https://mlsanity.com/paper/2609.27615/) (2026)

## Abstract (arXiv)

Multimodal emotion recognition (MER), leveraging speech and text, has emerged as a pivotal domain within human-computer interaction, demanding sophisticated methods for effective multimodal integration. The challenge of aligning features across these modalities is significant, with most existing approaches adopting a singular alignment strategy. Such a narrow focus not only limits model performance but also fails to address the complexity and ambiguity inherent in emotional expressions. In response, this paper introduces a Multi-Granularity Cross-Modal Alignment (MGCMA) framework, distinguished by its comprehensive approach encompassing distribution-based, instance-based, and token-based alignment modules. This framework enables a multi-level perception of emotional information across modalities. Our experiments on IEMOCAP demonstrate that our proposed method outperforms current state-of-the-art techniques.