# MAVEN: Multi-modal Attention for Valence-Arousal Emotion Network

arXiv: https://arxiv.org/abs/2503.12623 · HTML: https://mlsanity.com/paper/2503.12623/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 104 references that are in the map.
- First posted on arXiv 16 Mar 2025.

## Builds on (most cited first, 30 of 47)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks](https://mlsanity.com/paper/2208.10442/) (2022)
- [Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFace](https://mlsanity.com/paper/1910.04855/) (2019)
- [Deep Affect Prediction in-the-wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyond](https://mlsanity.com/paper/1804.10938/) (2018)
- [Analysing Affective Behavior in the First ABAW 2020 Competition](https://mlsanity.com/paper/2001.11409/) (2020)
- [Affect Analysis in-the-wild: Valence-Arousal, Expressions, Action Units and a Unified Framework](https://mlsanity.com/paper/2103.15792/) (2021)
- [Distribution Matching for Heterogeneous Multi-Task Learning: a Large-scale Face Study](https://mlsanity.com/paper/2105.03790/) (2021)
- [Analysing Affective Behavior in the second ABAW2 Competition](https://mlsanity.com/paper/2106.15318/) (2021)
- [ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Multi-Task Learning Challenges](https://mlsanity.com/paper/2202.10659/) (2022)
- [Face Behavior a la carte: Expressions, Affect and Action Units in a Single Network](https://mlsanity.com/paper/1910.11111/) (2019)
- [ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Emotional Reaction Intensity Estimation Challenges](https://mlsanity.com/paper/2303.01498/) (2023)
- [Aff-Wild2: Extending the Aff-Wild Database for Affect Recognition](https://mlsanity.com/paper/1811.07770/) (2018)
- [The 6th Affective Behavior Analysis in-the-wild (ABAW) Competition](https://mlsanity.com/paper/2402.19344/) (2024)
- [Multi-Head Attention: Collaborate Instead of Concatenate](https://mlsanity.com/paper/2006.16362/) (2020)
- [Distribution Matching for Multi-Task Learning of Classification Tasks: a Large-Scale Study on Faces & Beyond](https://mlsanity.com/paper/2401.01219/) (2024)
- [7th ABAW Competition: Multi-Task Learning and Compound Expression Recognition](https://mlsanity.com/paper/2407.03835/) (2024)
- [Two-Stream Aural-Visual Affect Analysis in the Wild](https://mlsanity.com/paper/2002.03399/) (2020)
- [Exploiting multi-CNN features in CNN-RNN based Dimensional Emotion Recognition on the OMG in-the-wild Dataset](https://mlsanity.com/paper/1910.01417/) (2019)
- [A Joint Cross-Attention Model for Audio-Visual Fusion in Dimensional Emotion Recognition](https://mlsanity.com/paper/2203.14779/) (2022)
- [EmotiCrafter: Text-to-Emotional-Image Generation based on Valence-Arousal Model](https://mlsanity.com/paper/2501.05710/) (2025)
- [A Multi-Task Learning & Generation Framework: Valence-Arousal, Action Units & Primary Expressions](https://mlsanity.com/paper/1811.07771/) (2018)
- [Interpretable Spatio-temporal Attention for Video Action Recognition](https://mlsanity.com/paper/1810.04511/) (2018)
- [Affective Behaviour Analysis via Integrating Multi-Modal Knowledge](https://mlsanity.com/paper/2403.10825/) (2024)
- [MIMAMO Net: Integrating Micro- and Macro-motion for Video Emotion Recognition](https://mlsanity.com/paper/1911.09784/) (2019)
- [Frame-level Prediction of Facial Expressions, Valence, Arousal and Action Units for Mobile Devices](https://mlsanity.com/paper/2203.13436/) (2022)

## Built on by (most cited first, 8 of 8)

- [EmoSpace: Immersive Affective Image Generation Guided by Fine-Grained Emotion Prototypes](https://mlsanity.com/paper/2602.11658/) (2026)
- [Stage-Adaptive Reliability Modeling for Continuous Valence-Arousal Estimation](https://mlsanity.com/paper/2603.11468/) (2026)
- [HSEmotion Team at ABAW-10 Competition: Facial Expression Recognition, Valence-Arousal Estimation, Action Unit Detection and Fine-Grained Violence Classification](https://mlsanity.com/paper/2603.12693/) (2026)
- [LaScA: Language-Conditioned Scalable Modelling of Affective Dynamics](https://mlsanity.com/paper/2604.07193/) (2026)
- [Modeling Induced Pleasure through Cognitive Appraisal Prediction via Multimodal Fusion](https://mlsanity.com/paper/2604.23753/) (2026)
- [Quality-Aware Multimodal Fusion Reveals Implicit Identity in Valence-Arousal Features](https://mlsanity.com/paper/2607.21347/) (2026)
- [Balancing Emotional Alignment and Semantic Consistency in Image Generation via Reinforcement Learning with Valence-Arousal Anchoring](https://mlsanity.com/paper/2609.12830/) (2026)
- [A Comprehensive Review of Multimodal Facial State Analysis: Tasks, Methods, and Resources](https://mlsanity.com/paper/2609.13255/) (2026)

## Abstract (arXiv)

Dynamic emotion recognition in the wild remains challenging due to the transient nature of emotional expressions and temporal misalignment of multi-modal cues. Traditional approaches predict valence and arousal and often overlook the inherent correlation between these two dimensions. The proposed Multi-modal Attention for Valence-Arousal Emotion Network (MAVEN) integrates visual, audio, and textual modalities through a bi-directional cross-modal attention mechanism. MAVEN uses modality-specific encoders to extract features from synchronized video frames, audio segments, and transcripts, predicting emotions in polar coordinates following Russell's circumplex model. The evaluation of the Aff-Wild2 dataset using MAVEN achieved a concordance correlation coefficient (CCC) of 0.3061, surpassing the ResNet-50 baseline model with a CCC of 0.22. The multistage architecture captures the subtle and transient nature of emotional expressions in conversational videos and improves emotion recognition in real-world situations. The code is available at: https://github.com/Vrushank-Ahire/MAVEN_8th_ABAW