# Solution for 10th Competition on Ambivalence/Hesitancy (AH) Video Recognition Challenge using Divergence-Based Multimodal Fusion

arXiv: https://arxiv.org/abs/2603.16939 · HTML: https://mlsanity.com/paper/2603.16939/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 32 references that are in the map.
- First posted on arXiv 15 Mar 2026.

## Builds on (most cited first, 24 of 24)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [MediaPipe: A Framework for Building Perception Pipelines](https://mlsanity.com/paper/1906.08172/) (2019)
- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [GoEmotions: A Dataset of Fine-Grained Emotions](https://mlsanity.com/paper/2005.00547/) (2020)
- [Attention Bottlenecks for Multimodal Fusion](https://mlsanity.com/paper/2107.00135/) (2021)
- [Deep Facial Expression Recognition: A Survey](https://mlsanity.com/paper/1804.08348/) (2018)
- [Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFace](https://mlsanity.com/paper/1910.04855/) (2019)
- [Dawn of the transformer era in speech emotion recognition: closing the valence gap](https://mlsanity.com/paper/2203.07378/) (2022)
- [The 6th Affective Behavior Analysis in-the-wild (ABAW) Competition](https://mlsanity.com/paper/2402.19344/) (2024)
- [End-to-End Multimodal Emotion Recognition using Deep Neural Networks](https://mlsanity.com/paper/1704.08619/) (2017)
- [JhatAA-Net: Joint Facial Action Unit Detection and Face Alignment via Adaptive Attention](https://mlsanity.com/paper/2003.08834/) (2020)
- [Py-Feat: Python Facial Expression Analysis Toolbox](https://mlsanity.com/paper/2104.03509/) (2021)
- [Continuous Emotion Recognition using Visual-audio-linguistic information: A Technical Report for ABAW3](https://mlsanity.com/paper/2203.13031/) (2022)
- [HSEmotion Team at ABAW-8 Competition: Audiovisual Ambivalence/Hesitancy, Emotional Mimicry Intensity and Facial Expression Recognition](https://mlsanity.com/paper/2503.10399/) (2025)
- [BAH Dataset for Ambivalence/Hesitancy Recognition in Videos for Digital Behavioural Change](https://mlsanity.com/paper/2505.19328/) (2025)
- [Textualized and Feature-based Models for Compound Multimodal Emotion Recognition in the Wild](https://mlsanity.com/paper/2407.12927/) (2024)
- [Efficient Feature Extraction and Late Fusion Strategy for Audiovisual Emotional Mimicry Intensity Estimation](https://mlsanity.com/paper/2403.11757/) (2024)
- [Unimodal Multi-Task Fusion for Emotional Mimicry Intensity Prediction](https://mlsanity.com/paper/2403.11879/) (2024)
- [Semantic Matters: Multimodal Features for Affective Analysis](https://mlsanity.com/paper/2504.11460/) (2025)

## Built on by (most cited first, 4 of 4)

- [From Affect to Complex Behavior: Advancing Multimodal Human-Centered AI at the 10th ABAW Workshop & Competition](https://mlsanity.com/paper/2605.27451/) (2026)
- [HEDGE: A Calibrated Ensemble for A/H Recognition](https://mlsanity.com/paper/2607.12176/) (2026)
- [CF-Net: Conflict Fusion with Speaker Normalisation and Certainty Weighting for Ambivalence/Hesitancy Recognition](https://mlsanity.com/paper/2607.13976/) (2026)
- [Multimodal Ambivalence and Hesitancy Recognition via Cross-Attention and Gated Fusion](https://mlsanity.com/paper/2607.15779/) (2026)

## Abstract (arXiv)

We address the Ambivalence/Hesitancy (A/H) Video Recognition Challenge at the 10th ABAW Competition (CVPR 2026). We propose a divergence-based multimodal fusion that explicitly measures cross-modal conflict between visual, audio, and textual channels. Visual features are encoded as Action Units (AUs) extracted via Py-Feat, audio via Wav2Vec 2.0, and text via BERT. Each modality is processed by a BiLSTM with attention pooling and projected into a shared embedding space. The fusion module computes pairwise absolute differences between modality embeddings, directly capturing the incongruence that characterizes A/H. On the BAH dataset, our approach achieves a Macro F1 of 0.6808 on the validation test set, outperforming the challenge baseline of 0.2827. Statistical analysis across 1{,}132 videos confirms that temporal variability of AUs is the dominant visual discriminator of A/H.