# Cross-Attention is Not Always Needed: Dynamic Cross-Attention for Audio-Visual Dimensional Emotion Recognition

arXiv: https://arxiv.org/abs/2403.19554 · HTML: https://mlsanity.com/paper/2403.19554/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 26 references that are in the map.
- First posted on arXiv 28 Mar 2024.

## Builds on (most cited first, 16 of 16)

- [Multimodal Learning with Transformers: A Survey](https://mlsanity.com/paper/2206.06488/) (2022)
- [ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Emotional Reaction Intensity Estimation Challenges](https://mlsanity.com/paper/2303.01498/) (2023)
- [End-to-End Multimodal Emotion Recognition using Deep Neural Networks](https://mlsanity.com/paper/1704.08619/) (2017)
- [Regularizing Deep Neural Networks by Noise: Its Interpretation and Optimization](https://mlsanity.com/paper/1710.05179/) (2017)
- [Leveraging Recent Advances in Deep Learning for Audio-Visual Emotion Recognition](https://mlsanity.com/paper/2103.09154/) (2021)
- [M2Lens: Visualizing and Explaining Multimodal Models for Sentiment Analysis](https://mlsanity.com/paper/2107.08264/) (2021)
- [Multi-modal Facial Affective Analysis based on Masked Autoencoder](https://mlsanity.com/paper/2303.10849/) (2023)
- [Leveraging TCN and Transformer for effective visual-audio fusion in continuous emotion recognition](https://mlsanity.com/paper/2303.08356/) (2023)
- [Cross Attentional Audio-Visual Fusion for Dimensional Emotion Recognition](https://mlsanity.com/paper/2111.05222/) (2021)
- [Audio-Visual Fusion for Emotion Recognition in the Valence-Arousal Space Using Joint Cross-Attention](https://mlsanity.com/paper/2209.09068/) (2022)
- [Gated Mechanism for Attention Based Multimodal Sentiment Analysis](https://mlsanity.com/paper/2003.01043/) (2020)
- [Group Gated Fusion on Attention-based Bidirectional Alignment for Multimodal Emotion Recognition](https://mlsanity.com/paper/2201.06309/) (2022)
- [Continuous Emotion Recognition with Audio-visual Leader-follower Attentive Fusion](https://mlsanity.com/paper/2107.01175/) (2021)
- [Detecting expressions with multimodal transformers](https://mlsanity.com/paper/2012.00063/) (2020)
- [Not All Attention Is Needed: Gated Attention Network for Sequence Data](https://mlsanity.com/paper/1912.00349/) (2019)
- [Emotion Recognition Using Fusion of Audio and Video Features](https://mlsanity.com/paper/1906.10623/) (2019)

## Built on by (most cited first, 3 of 3)

- [Attention-Driven Multimodal Alignment for Long-term Action Quality Assessment](https://mlsanity.com/paper/2507.21945/) (2025)
- [Stage-Adaptive Reliability Modeling for Continuous Valence-Arousal Estimation](https://mlsanity.com/paper/2603.11468/) (2026)
- [United we stand, Divided we fall: Handling Weak Complementary Relationships for Audio-Visual Emotion Recognition in Valence-Arousal Space](https://mlsanity.com/paper/2503.12261/) (2025)

## Abstract (arXiv)

In video-based emotion recognition, audio and visual modalities are often expected to have a complementary relationship, which is widely explored using cross-attention. However, they may also exhibit weak complementary relationships, resulting in poor representations of audio-visual features, thus degrading the performance of the system. To address this issue, we propose Dynamic Cross-Attention (DCA) that can dynamically select cross-attended or unattended features on the fly based on their strong or weak complementary relationship with each other, respectively. Specifically, a simple yet efficient gating layer is designed to evaluate the contribution of the cross-attention mechanism and choose cross-attended features only when they exhibit a strong complementary relationship, otherwise unattended features. We evaluate the performance of the proposed approach on the challenging RECOLA and Aff-Wild2 datasets. We also compare the proposed approach with other variants of cross-attention and show that the proposed model consistently improves the performance on both datasets.