mlsanityData to 7 Oct 2026

Enhancing Multimodal Emotion Recognition through Multi-Granularity Cross-Modal Alignment

arXiv 2412.20821 ↗First posted 30 Dec 2024eess.AS cs.CL cs.SDcited by 5

Not on the map. Why?

Abstract (arXiv)

Multimodal emotion recognition (MER), leveraging speech and text, has emerged as a pivotal domain within human-computer interaction, demanding sophisticated methods for effective multimodal integration. The challenge of aligning features across these modalities is significant, with most existing approaches adopting a singular alignment strategy. Such a narrow focus not only limits model performance but also fails to address the complexity and ambiguity inherent in emotional expressions. In response, this paper introduces a Multi-Granularity Cross-Modal Alignment (MGCMA) framework, distinguished by its comprehensive approach encompassing distribution-based, instance-based, and token-based alignment modules. This framework enables a multi-level perception of emotional information across modalities. Our experiments on IEMOCAP demonstrate that our proposed method outperforms current state-of-the-art techniques.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 14 of its 26 references that are in the map. First posted on arXiv 30 Dec 2024.

Builds on

14 of 26 references.

References · 14

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  4. 2022MAP: Multimodal Uncertainty-Aware Vision-Language Pre-training Modelcited by 41
  5. 2022Group Gated Fusion on Attention-based Bidirectional Alignment for Multimodal Emotion Recognitioncited by 16
  6. 2021Efficient Speech Emotion Recognition Using Multi-Scale CNN and Attentioncited by 13
  7. 2018Touch Your Heart: A Tone-aware Chatbot for Customer Care on Social Mediacited by 12
  8. 2023Fine-grained Disentangled Representation Learning for Multimodal Emotion Recognitioncited by 11
  9. 2020Learning Fine-Grained Cross Modality Excitement for Speech Emotion Recognitioncited by 10
  10. 2023Knowledge-aware Bayesian Co-attention for Multimodal Emotion Recognitioncited by 8
  11. 2023Using Auxiliary Tasks In Multimodal Fusion Of Wav2vec 2.0 And BERT For Multimodal Emotion Recognitioncited by 8
  12. 2020Multi-Classifier Interactive Learning for Ambiguous Speech Emotion Recognitioncited by 7
2 more
  1. 2018Convolutional Attention Networks for Multimodal Emotion Recognition from Speech and Text Datacited by 4
  2. 2023Supervised Contrastive Learning with Nearest Neighbor Search for Speech Emotion Recognitioncited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2025OmniVox: Zero-Shot Emotion Recognition with Omni-LLMscited by 7
  2. 2025Quality-Controlled Multimodal Emotion Recognition in Conversations with Identity-Based Transfer Learning and MAMBA Fusioncited by 0
  3. 2026EmoEUS: Uncertainty Supervision for Multimodal Emotion Recognition in Conversationcited by 0
  4. 2026Learning to Prefer Reliably: Error-Augmented Emotion Preference Optimization with Calibrated Fusioncited by 0
  5. 2026BiCFlow-MER: Orchestrating Discriminative and Generative Multimodal Emotion Recognition via Conditional Transportcited by 0