mlsanityData to 7 Oct 2026

Group Gated Fusion on Attention-based Bidirectional Alignment for Multimodal Emotion Recognition

arXiv 2201.06309 ↗First posted 17 Jan 2022cs.CL cs.SD eess.AScited by 16

Not on the map. Why?

Abstract (arXiv)

Emotion recognition is a challenging and actively-studied research area that plays a critical role in emotion-aware human-computer interaction systems. In a multimodal setting, temporal alignment between different modalities has not been well investigated yet. This paper presents a new model named as Gated Bidirectional Alignment Network (GBAN), which consists of an attention-based bidirectional alignment network over LSTM hidden states to explicitly capture the alignment relationship between speech and text, and a novel group gated fusion (GGF) layer to integrate the representations of different modalities. We empirically show that the attention-aligned representations outperform the last-hidden-states of LSTM significantly, and the proposed GBAN model outperforms existing state-of-the-art multimodal approaches on the IEMOCAP dataset.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 11 of its 28 references that are in the map. First posted on arXiv 17 Jan 2022.

Builds on

11 of 28 references.

References · 11

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Tensor Fusion Network for Multimodal Sentiment Analysiscited by 429
  3. 2017Gated Multimodal Units for Information Fusioncited by 244
  4. 2018Multimodal Speech Emotion Recognition Using Audio and Textcited by 55
  5. 2018Multimodal Sentiment Analysis using Hierarchical Fusion with Context Modelingcited by 39
  6. 2018Multimodal Sentiment Analysis: Addressing Key Issues and Setting up the Baselinescited by 23
  7. 2019Learning Alignment for Multimodal Emotion Recognition from Speechcited by 22
  8. 2017Attentive Convolutional Neural Network based Speech Emotion Recognition: A Study on the Impact of Input Features, Signal Length, and Acted Speechcited by 18
  9. 2019Deep neural networks for emotion recognition combining audio and transcriptscited by 11
  10. 2019Multimodal and Multi-view Models for Emotion Recognitioncited by 10
  11. 2019Bimodal Speech Emotion Recognition Using Pre-Trained Language Modelscited by 5

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 2; 2024: 7; 2025: 4.

16 papers.

Sort:

By area:

  1. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  2. 2023Multimodal Explainable Artificial Intelligence: A Comprehensive Review of Methodological Advances and Future Research Directionscited by 25
  3. 2024Masked Graph Learning with Recurrent Alignment for Multimodal Emotion Recognition in Conversationcited by 19
  4. 2022Multi-level Fusion of Wav2vec 2.0 and BERT for Multimodal Emotion Recognitioncited by 9
  5. 2022Interpretability for Multimodal Emotion Recognition using Concept Activation Vectorscited by 8
  6. 2022Estimating the Uncertainty in Emotion Class Labels with Utterance-Specific Dirichlet Priorscited by 8
  7. 2024MF-AED-AEC: Speech Emotion Recognition by Leveraging Multimodal Fusion, Asr Error Detection, and Asr Error Correctioncited by 8
  8. 2024Enhancing Multimodal Emotion Recognition through Multi-Granularity Cross-Modal Alignmentcited by 5
  9. 2024Cross-Attention is Not Always Needed: Dynamic Cross-Attention for Audio-Visual Dimensional Emotion Recognitioncited by 3
  10. 2024Turbo your multi-modal classification with contrastive learningcited by 2
  11. 2025GIA-MIC: Multimodal Emotion Recognition with Gated Interactive Attention and Modality-Invariant Learning Constraintscited by 1
  12. 2025Arabic Multimodal Machine Learning: Datasets, Applications, Approaches, and Challengescited by 1
4 more
  1. 2023On the Efficacy and Noise-Robustness of Jointly Learned Speech Emotion and Automatic Speech Recognitioncited by 0
  2. 2024Emotion Loss Attacking: Adversarial Attack Perception for Skeleton based on Multi-dimensional Featurescited by 0
  3. 2025DRKF: Decoupled Representations with Knowledge Fusion for Multimodal Emotion Recognitioncited by 0
  4. 2025Centering Emotion Hotspots: Multimodal Local-Global Fusion and Cross-Modal Alignment for Emotion Recognition in Conversationscited by 0