# GatedxLSTM: A Multimodal Affective Computing Approach for Emotion Recognition in Conversations

arXiv: https://arxiv.org/abs/2503.20919 · HTML: https://mlsanity.com/paper/2503.20919/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 57 references that are in the map.
- First posted on arXiv 26 Mar 2025.

## Builds on (most cited first, 19 of 19)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [xLSTM: Extended Long Short-Term Memory](https://mlsanity.com/paper/2405.04517/) (2024)
- [Emotion Recognition in Conversation: Research Challenges, Datasets, and Recent Advances](https://mlsanity.com/paper/1905.02947/) (2019)
- [Revisiting Multimodal Emotion Recognition in Conversation from the Perspective of Graph Spectrum](https://mlsanity.com/paper/2404.17862/) (2024)
- [Attention Based Fully Convolutional Network for Speech Emotion Recognition](https://mlsanity.com/paper/1806.01506/) (2018)
- [Recent Advances in Multimodal Affective Computing: An NLP Perspective](https://mlsanity.com/paper/2409.07388/) (2024)
- [Bio-xLSTM: Generative modeling, representation and in-context learning of biological and chemical sequences](https://mlsanity.com/paper/2411.04165/) (2024)
- [BiosERC: Integrating Biography Speakers Supported by LLMs for ERC Tasks](https://mlsanity.com/paper/2407.04279/) (2024)
- [GEmo-CLAP: Gender-Attribute-Enhanced Contrastive Language-Audio Pretraining for Accurate Speech Emotion Recognition](https://mlsanity.com/paper/2306.07848/) (2023)
- [ParaCLAP -- Towards a general language-audio model for computational paralinguistic tasks](https://mlsanity.com/paper/2406.07203/) (2024)
- [LLM supervised Pre-training for Multimodal Emotion Recognition in Conversations](https://mlsanity.com/paper/2501.11468/) (2025)
- [Multimodal Emotion Recognition using Transfer Learning from Speaker Recognition and BERT-based models](https://mlsanity.com/paper/2202.08974/) (2022)
- [Multimodal Prompt Transformer with Hybrid Contrastive Learning for Emotion Recognition in Conversation](https://mlsanity.com/paper/2310.04456/) (2023)
- [Testing Correctness, Fairness, and Robustness of Speech Emotion Recognition Models](https://mlsanity.com/paper/2312.06270/) (2023)
- [Distribution-based Emotion Recognition in Conversation](https://mlsanity.com/paper/2211.04834/) (2022)
- [Multimodal Emotion Recognition from Raw Audio with Sinc-convolution](https://mlsanity.com/paper/2402.11954/) (2024)
- [FATRER: Full-Attention Topic Regularizer for Accurate and Robust Conversational Emotion Recognition](https://mlsanity.com/paper/2307.12221/) (2023)
- [xLSTM-FER: Enhancing Student Expression Recognition with Extended Vision Long Short-Term Memory Network](https://mlsanity.com/paper/2410.05074/) (2024)

## Built on by (most cited first, 3 of 3)

- [Dynamic Parameter Memory: Temporary LoRA-Enhanced LLM for Long-Sequence Emotion Recognition in Conversation](https://mlsanity.com/paper/2507.09076/) (2025)
- [GateMABSA: Aspect-Image Gated Fusion for Multimodal Aspect-based Sentiment Analysis](https://mlsanity.com/paper/2509.25037/) (2025)
- [Explainable Speech Emotion Recognition: Weighted Attribute Fairness to Model Demographic Contributions to Social Bias](https://mlsanity.com/paper/2604.19763/) (2026)

## Abstract (arXiv)

Affective Computing (AC) is essential for advancing Artificial General Intelligence (AGI), with emotion recognition serving as a key component. However, human emotions are inherently dynamic, influenced not only by an individual's expressions but also by interactions with others, and single-modality approaches often fail to capture their full dynamics. Multimodal Emotion Recognition (MER) leverages multiple signals but traditionally relies on utterance-level analysis, overlooking the dynamic nature of emotions in conversations. Emotion Recognition in Conversation (ERC) addresses this limitation, yet existing methods struggle to align multimodal features and explain why emotions evolve within dialogues. To bridge this gap, we propose GatedxLSTM, a novel speech-text multimodal ERC model that explicitly considers voice and transcripts of both the speaker and their conversational partner(s) to identify the most influential sentences driving emotional shifts. By integrating Contrastive Language-Audio Pretraining (CLAP) for improved cross-modal alignment and employing a gating mechanism to emphasise emotionally impactful utterances, GatedxLSTM enhances both interpretability and performance. Additionally, the Dialogical Emotion Decoder (DED) refines emotion predictions by modelling contextual dependencies. Experiments on the IEMOCAP dataset demonstrate that GatedxLSTM achieves state-of-the-art (SOTA) performance among open-source methods in four-class emotion classification. These results validate its effectiveness for ERC applications and provide an interpretability analysis from a psychological perspective.