# Mamba-Enhanced Text-Audio-Video Alignment Network for Emotion Recognition in Conversations

arXiv: https://arxiv.org/abs/2409.05243 · HTML: https://mlsanity.com/paper/2409.05243/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Speech and audio SSMs, one of 10 directions in State-space sequence models (S4, Mamba).
- Builds on 12 of its 27 references that are in the map.
- First posted on arXiv 8 Sep 2024.

## Placed in

- State-space sequence models (S4, Mamba) / [Speech and audio SSMs](https://mlsanity.com/ssm/speech-and-audio-ssms/)

## Builds on (most cited first, 12 of 12)

- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations](https://mlsanity.com/paper/1810.02508/) (2018)
- [MMGCN: Multimodal Fusion via Deep Graph Convolution Network for Emotion Recognition in Conversation](https://mlsanity.com/paper/2107.06779/) (2021)
- [EmoBERTa: Speaker-Aware Emotion Recognition in Conversation with RoBERTa](https://mlsanity.com/paper/2108.12009/) (2021)
- [Deep Imbalanced Learning for Multimodal Emotion Recognition in Conversations](https://mlsanity.com/paper/2312.06337/) (2023)
- [Emotion Detection for Misinformation: A Review](https://mlsanity.com/paper/2311.00671/) (2023)
- [GA2MIF: Graph and Attention Based Two-Stage Multi-Source Information Fusion for Conversational Emotion Detection](https://mlsanity.com/paper/2207.11900/) (2022)
- [Masked Graph Learning with Recurrent Alignment for Multimodal Emotion Recognition in Conversation](https://mlsanity.com/paper/2407.16714/) (2024)
- [Multimodal Dual Emotion with Fusion of Visual Sentiment for Rumor Detection](https://mlsanity.com/paper/2204.11515/) (2022)

## Built on by (most cited first, 3 of 3)

- [Mixture-of-Shape-Experts (MoSE): End-to-End Shape Dictionary Framework to Prompt SAM for Generalizable Medical Segmentation](https://mlsanity.com/paper/2504.09601/) (2025)
- [DA-Mamba: Dialogue-aware selective state-space model for multimodal engagement estimation](https://mlsanity.com/paper/2509.17711/) (2025)
- [Emotion as a Distribution: Joint Valence-Arousal Probability Learning for Speaker-Independent Multimodal Emotion Recognition](https://mlsanity.com/paper/2609.05755/) (2026)

## Abstract (arXiv)

Emotion Recognition in Conversations (ERCs) is a vital area within multimodal interaction research, dedicated to accurately identifying and classifying the emotions expressed by speakers throughout a conversation. Traditional ERC approaches predominantly rely on unimodal cues-such as text, audio, or visual data-leading to limitations in their effectiveness. These methods encounter two significant challenges: 1) Consistency in multimodal information. Before integrating various modalities, it is crucial to ensure that the data from different sources is aligned and coherent. 2) Contextual information capture. Successfully fusing multimodal features requires a keen understanding of the evolving emotional tone, especially in lengthy dialogues where emotions may shift and develop over time. To address these limitations, we propose a novel Mamba-enhanced Text-Audio-Video alignment network (MaTAV) for the ERC task. MaTAV is with the advantages of aligning unimodal features to ensure consistency across different modalities and handling long input sequences to better capture contextual multimodal information. The extensive experiments on the MELD and IEMOCAP datasets demonstrate that MaTAV significantly outperforms existing state-of-the-art methods on the ERC task with a big margin.