# Cross Attentional Audio-Visual Fusion for Dimensional Emotion Recognition

arXiv: https://arxiv.org/abs/2111.05222 · HTML: https://mlsanity.com/paper/2111.05222/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 43 references that are in the map.
- First posted on arXiv 9 Nov 2021.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks](https://mlsanity.com/paper/1604.02878/) (2016)
- [Cross Attention Network for Few-shot Classification](https://mlsanity.com/paper/1910.07677/) (2019)
- [End-to-End Multimodal Emotion Recognition using Deep Neural Networks](https://mlsanity.com/paper/1704.08619/) (2017)
- [AVEC 2016 - Depression, Mood, and Emotion Recognition Workshop and Challenge](https://mlsanity.com/paper/1605.01600/) (2016)
- [Weakly-Supervised Action Localization by Generative Attention Modeling](https://mlsanity.com/paper/2003.12424/) (2020)
- [Emotion Recognition in Speech using Cross-Modal Transfer in the Wild](https://mlsanity.com/paper/1808.05561/) (2018)
- [Leveraging Recent Advances in Deep Learning for Audio-Visual Emotion Recognition](https://mlsanity.com/paper/2103.09154/) (2021)
- [An End-to-End Visual-Audio Attention Network for Emotion Recognition in User-Generated Videos](https://mlsanity.com/paper/2003.00832/) (2020)
- [Visual Question Answering with Memory-Augmented Networks](https://mlsanity.com/paper/1707.04968/) (2017)
- [Deep Multimodal Representation Learning from Temporal Data](https://mlsanity.com/paper/1704.03152/) (2017)
- [Detecting expressions with multimodal transformers](https://mlsanity.com/paper/2012.00063/) (2020)
- [Deep Auto-Encoders with Sequential Learning for Multimodal Dimensional Emotion Recognition](https://mlsanity.com/paper/2004.13236/) (2020)
- [Emotion Recognition Using Fusion of Audio and Video Features](https://mlsanity.com/paper/1906.10623/) (2019)

## Built on by (most cited first, 19 of 19)

- [A Joint Cross-Attention Model for Audio-Visual Fusion in Dimensional Emotion Recognition](https://mlsanity.com/paper/2203.14779/) (2022)
- [Audio-Visual Fusion for Emotion Recognition in the Valence-Arousal Space Using Joint Cross-Attention](https://mlsanity.com/paper/2209.09068/) (2022)
- [Recursive Joint Cross-Modal Attention for Multimodal Fusion in Dimensional Emotion Recognition](https://mlsanity.com/paper/2403.13659/) (2024)
- [Joint Multimodal Transformer for Emotion Recognition in the Wild](https://mlsanity.com/paper/2403.10488/) (2024)
- [BAH Dataset for Ambivalence/Hesitancy Recognition in Videos for Digital Behavioural Change](https://mlsanity.com/paper/2505.19328/) (2025)
- [Textualized and Feature-based Models for Compound Multimodal Emotion Recognition in the Wild](https://mlsanity.com/paper/2407.12927/) (2024)
- [MultiEYE: Dataset and Benchmark for OCT-Enhanced Retinal Disease Recognition from Fundus Images](https://mlsanity.com/paper/2412.09402/) (2024)
- [Recursive Joint Attention for Audio-Visual Fusion in Regression based Emotion Recognition](https://mlsanity.com/paper/2304.07958/) (2023)
- [Audio-Visual Speaker Verification via Joint Cross-Attention](https://mlsanity.com/paper/2309.16569/) (2023)
- [Video-based Cross-modal Auxiliary Network for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2208.13954/) (2022)
- [Audio-Visual Person Verification based on Recursive Fusion of Joint Cross-Attention](https://mlsanity.com/paper/2403.04654/) (2024)
- [Dynamic Cross Attention for Audio-Visual Person Verification](https://mlsanity.com/paper/2403.04661/) (2024)
- [Cross-Attention is Not Always Needed: Dynamic Cross-Attention for Audio-Visual Dimensional Emotion Recognition](https://mlsanity.com/paper/2403.19554/) (2024)
- [Multimodal Ambivalence/Hesitancy Recognition in Videos for Personalized Digital Health Interventions](https://mlsanity.com/paper/2604.11730/) (2026)
- [Improving the Modality Representation with Multi-View Contrastive Learning for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2210.15824/) (2022)
- [FV2ES: A Fully End2End Multimodal System for Fast Yet Effective Video Emotion Recognition Inference](https://mlsanity.com/paper/2209.10170/) (2022)
- [Inconsistency-Aware Cross-Attention for Audio-Visual Fusion in Dimensional Emotion Recognition](https://mlsanity.com/paper/2405.12853/) (2024)
- [Interactive Multimodal Fusion with Temporal Modeling](https://mlsanity.com/paper/2503.10523/) (2025)
- [United we stand, Divided we fall: Handling Weak Complementary Relationships for Audio-Visual Emotion Recognition in Valence-Arousal Space](https://mlsanity.com/paper/2503.12261/) (2025)

## Abstract (arXiv)

Multimodal analysis has recently drawn much interest in affective computing, since it can improve the overall accuracy of emotion recognition over isolated uni-modal approaches. The most effective techniques for multimodal emotion recognition efficiently leverage diverse and complimentary sources of information, such as facial, vocal, and physiological modalities, to provide comprehensive feature representations. In this paper, we focus on dimensional emotion recognition based on the fusion of facial and vocal modalities extracted from videos, where complex spatiotemporal relationships may be captured. Most of the existing fusion techniques rely on recurrent networks or conventional attention mechanisms that do not effectively leverage the complimentary nature of audio-visual (A-V) modalities. We introduce a cross-attentional fusion approach to extract the salient features across A-V modalities, allowing for accurate prediction of continuous values of valence and arousal. Our new cross-attentional A-V fusion model efficiently leverages the inter-modal relationships. In particular, it computes cross-attention weights to focus on the more contributive features across individual modalities, and thereby combine contributive feature representations, which are then fed to fully connected layers for the prediction of valence and arousal. The effectiveness of the proposed approach is validated experimentally on videos from the RECOLA and Fatigue (private) data-sets. Results indicate that our cross-attentional A-V fusion model is a cost-effective approach that outperforms state-of-the-art fusion approaches. Code is available: urlhttps://github.com/praveena2j/Cross-Attentional-AV-Fusion