# Group Gated Fusion on Attention-based Bidirectional Alignment for Multimodal Emotion Recognition

arXiv: https://arxiv.org/abs/2201.06309 · HTML: https://mlsanity.com/paper/2201.06309/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 28 references that are in the map.
- First posted on arXiv 17 Jan 2022.

## Builds on (most cited first, 11 of 11)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Tensor Fusion Network for Multimodal Sentiment Analysis](https://mlsanity.com/paper/1707.07250/) (2017)
- [Gated Multimodal Units for Information Fusion](https://mlsanity.com/paper/1702.01992/) (2017)
- [Multimodal Speech Emotion Recognition Using Audio and Text](https://mlsanity.com/paper/1810.04635/) (2018)
- [Multimodal Sentiment Analysis using Hierarchical Fusion with Context Modeling](https://mlsanity.com/paper/1806.06228/) (2018)
- [Multimodal Sentiment Analysis: Addressing Key Issues and Setting up the Baselines](https://mlsanity.com/paper/1803.07427/) (2018)
- [Learning Alignment for Multimodal Emotion Recognition from Speech](https://mlsanity.com/paper/1909.05645/) (2019)
- [Attentive Convolutional Neural Network based Speech Emotion Recognition: A Study on the Impact of Input Features, Signal Length, and Acted Speech](https://mlsanity.com/paper/1706.00612/) (2017)
- [Deep neural networks for emotion recognition combining audio and transcripts](https://mlsanity.com/paper/1911.00432/) (2019)
- [Multimodal and Multi-view Models for Emotion Recognition](https://mlsanity.com/paper/1906.10198/) (2019)
- [Bimodal Speech Emotion Recognition Using Pre-Trained Language Models](https://mlsanity.com/paper/1912.02610/) (2019)

## Built on by (most cited first, 16 of 16)

- [Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2412.02104/) (2024)
- [Multimodal Explainable Artificial Intelligence: A Comprehensive Review of Methodological Advances and Future Research Directions](https://mlsanity.com/paper/2306.05731/) (2023)
- [Masked Graph Learning with Recurrent Alignment for Multimodal Emotion Recognition in Conversation](https://mlsanity.com/paper/2407.16714/) (2024)
- [Multi-level Fusion of Wav2vec 2.0 and BERT for Multimodal Emotion Recognition](https://mlsanity.com/paper/2207.04697/) (2022)
- [Interpretability for Multimodal Emotion Recognition using Concept Activation Vectors](https://mlsanity.com/paper/2202.01072/) (2022)
- [Estimating the Uncertainty in Emotion Class Labels with Utterance-Specific Dirichlet Priors](https://mlsanity.com/paper/2203.04443/) (2022)
- [MF-AED-AEC: Speech Emotion Recognition by Leveraging Multimodal Fusion, Asr Error Detection, and Asr Error Correction](https://mlsanity.com/paper/2401.13260/) (2024)
- [Enhancing Multimodal Emotion Recognition through Multi-Granularity Cross-Modal Alignment](https://mlsanity.com/paper/2412.20821/) (2024)
- [Cross-Attention is Not Always Needed: Dynamic Cross-Attention for Audio-Visual Dimensional Emotion Recognition](https://mlsanity.com/paper/2403.19554/) (2024)
- [Turbo your multi-modal classification with contrastive learning](https://mlsanity.com/paper/2409.09282/) (2024)
- [GIA-MIC: Multimodal Emotion Recognition with Gated Interactive Attention and Modality-Invariant Learning Constraints](https://mlsanity.com/paper/2506.00865/) (2025)
- [Arabic Multimodal Machine Learning: Datasets, Applications, Approaches, and Challenges](https://mlsanity.com/paper/2508.12227/) (2025)
- [On the Efficacy and Noise-Robustness of Jointly Learned Speech Emotion and Automatic Speech Recognition](https://mlsanity.com/paper/2305.12540/) (2023)
- [Emotion Loss Attacking: Adversarial Attack Perception for Skeleton based on Multi-dimensional Features](https://mlsanity.com/paper/2406.19815/) (2024)
- [DRKF: Decoupled Representations with Knowledge Fusion for Multimodal Emotion Recognition](https://mlsanity.com/paper/2508.01644/) (2025)
- [Centering Emotion Hotspots: Multimodal Local-Global Fusion and Cross-Modal Alignment for Emotion Recognition in Conversations](https://mlsanity.com/paper/2510.08606/) (2025)

## Abstract (arXiv)

Emotion recognition is a challenging and actively-studied research area that plays a critical role in emotion-aware human-computer interaction systems. In a multimodal setting, temporal alignment between different modalities has not been well investigated yet. This paper presents a new model named as Gated Bidirectional Alignment Network (GBAN), which consists of an attention-based bidirectional alignment network over LSTM hidden states to explicitly capture the alignment relationship between speech and text, and a novel group gated fusion (GGF) layer to integrate the representations of different modalities. We empirically show that the attention-aligned representations outperform the last-hidden-states of LSTM significantly, and the proposed GBAN model outperforms existing state-of-the-art multimodal approaches on the IEMOCAP dataset.