Solution for 10th Competition on Ambivalence/Hesitancy (AH) Video Recognition Challenge using Divergence-Based Multimodal Fusion
Not on the map. Why?
Abstract (arXiv)
We address the Ambivalence/Hesitancy (A/H) Video Recognition Challenge at the 10th ABAW Competition (CVPR 2026). We propose a divergence-based multimodal fusion that explicitly measures cross-modal conflict between visual, audio, and textual channels. Visual features are encoded as Action Units (AUs) extracted via Py-Feat, audio via Wav2Vec 2.0, and text via BERT. Each modality is processed by a BiLSTM with attention pooling and projected into a shared embedding space. The fusion module computes pairwise absolute differences between modality embeddings, directly capturing the incongruence that characterizes A/H. On the BAH dataset, our approach achieves a Macro F1 of 0.6808 on the validation test set, outperforming the challenge baseline of 0.2827. Statistical analysis across 1{,}132 videos confirms that temporal variability of AUs is the dominant visual discriminator of A/H.
Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 24 of its 32 references that are in the map. First posted on arXiv 15 Mar 2026.
Builds on
24 of 32 references.
References · 24
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
- 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
- 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
- 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
- 2019MediaPipe: A Framework for Building Perception Pipelinescited by 572
- 2017AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wildcited by 490
- 2020GoEmotions: A Dataset of Fine-Grained Emotionscited by 479
- 2021Attention Bottlenecks for Multimodal Fusioncited by 361
- 2018Deep Facial Expression Recognition: A Surveycited by 269
- 2019Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFacecited by 222
12 more
- 2022Dawn of the transformer era in speech emotion recognition: closing the valence gapcited by 155
- 2024The 6th Affective Behavior Analysis in-the-wild (ABAW) Competitioncited by 78
- 2017End-to-End Multimodal Emotion Recognition using Deep Neural Networkscited by 69
- 2020JhatAA-Net: Joint Facial Action Unit Detection and Face Alignment via Adaptive Attentioncited by 62
- 2021Py-Feat: Python Facial Expression Analysis Toolboxcited by 33
- 2022Continuous Emotion Recognition using Visual-audio-linguistic information: A Technical Report for ABAW3cited by 20
- 2025HSEmotion Team at ABAW-8 Competition: Audiovisual Ambivalence/Hesitancy, Emotional Mimicry Intensity and Facial Expression Recognitioncited by 15
- 2025BAH Dataset for Ambivalence/Hesitancy Recognition in Videos for Digital Behavioural Changecited by 14
- 2024Textualized and Feature-based Models for Compound Multimodal Emotion Recognition in the Wildcited by 9
- 2024Efficient Feature Extraction and Late Fusion Strategy for Audiovisual Emotional Mimicry Intensity Estimationcited by 6
- 2024Unimodal Multi-Task Fusion for Emotional Mimicry Intensity Predictioncited by 6
- 2025Semantic Matters: Multimodal Features for Affective Analysiscited by 5
Built on by
Citing papers per year: 2026: 4.
4 papers.
- 2026From Affect to Complex Behavior: Advancing Multimodal Human-Centered AI at the 10th ABAW Workshop & Competitioncited by 12
- 2026HEDGE: A Calibrated Ensemble for A/H Recognitioncited by 0
- 2026CF-Net: Conflict Fusion with Speaker Normalisation and Certainty Weighting for Ambivalence/Hesitancy Recognitioncited by 0
- 2026Multimodal Ambivalence and Hesitancy Recognition via Cross-Attention and Gated Fusioncited by 0
- 2026Multimodal Ambivalence and Hesitancy Recognition via Cross-Attention and Gated Fusion17 Jul 2026
- 2026CF-Net: Conflict Fusion with Speaker Normalisation and Certainty Weighting for Ambivalence/Hesitancy Recognition15 Jul 2026
- 2026HEDGE: A Calibrated Ensemble for A/H Recognition13 Jul 2026
- 2026From Affect to Complex Behavior: Advancing Multimodal Human-Centered AI at the 10th ABAW Workshop & Competition24 May 2026