# HSEmotion Team at ABAW-8 Competition: Audiovisual Ambivalence/Hesitancy, Emotional Mimicry Intensity and Facial Expression Recognition

arXiv: https://arxiv.org/abs/2503.10399 · HTML: https://mlsanity.com/paper/2503.10399/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 49 references that are in the map.
- First posted on arXiv 13 Mar 2025.

## Builds on (most cited first, 26 of 26)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [MediaPipe: A Framework for Building Perception Pipelines](https://mlsanity.com/paper/1906.08172/) (2019)
- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [GoEmotions: A Dataset of Fine-Grained Emotions](https://mlsanity.com/paper/2005.00547/) (2020)
- [Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFace](https://mlsanity.com/paper/1910.04855/) (2019)
- [Deep Affect Prediction in-the-wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyond](https://mlsanity.com/paper/1804.10938/) (2018)
- [Analysing Affective Behavior in the First ABAW 2020 Competition](https://mlsanity.com/paper/2001.11409/) (2020)
- [Distribution Matching for Heterogeneous Multi-Task Learning: a Large-scale Face Study](https://mlsanity.com/paper/2105.03790/) (2021)
- [Analysing Affective Behavior in the second ABAW2 Competition](https://mlsanity.com/paper/2106.15318/) (2021)
- [ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Multi-Task Learning Challenges](https://mlsanity.com/paper/2202.10659/) (2022)
- [Face Behavior a la carte: Expressions, Affect and Action Units in a Single Network](https://mlsanity.com/paper/1910.11111/) (2019)
- [ABAW: Learning from Synthetic Data & Multi-Task Learning Challenges](https://mlsanity.com/paper/2207.01138/) (2022)
- [ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Emotional Reaction Intensity Estimation Challenges](https://mlsanity.com/paper/2303.01498/) (2023)
- [The 6th Affective Behavior Analysis in-the-wild (ABAW) Competition](https://mlsanity.com/paper/2402.19344/) (2024)
- [Distribution Matching for Multi-Task Learning of Classification Tasks: a Large-Scale Study on Faces & Beyond](https://mlsanity.com/paper/2401.01219/) (2024)
- [7th ABAW Competition: Multi-Task Learning and Compound Expression Recognition](https://mlsanity.com/paper/2407.03835/) (2024)
- [Multi-modal Facial Affective Analysis based on Masked Autoencoder](https://mlsanity.com/paper/2303.10849/) (2023)
- [Leveraging TCN and Transformer for effective visual-audio fusion in continuous emotion recognition](https://mlsanity.com/paper/2303.08356/) (2023)
- [Robust Light-Weight Facial Affective Behavior Recognition with CLIP](https://mlsanity.com/paper/2403.09915/) (2024)
- [Exploring Facial Expression Recognition through Semi-Supervised Pretraining and Temporal Modeling](https://mlsanity.com/paper/2403.11942/) (2024)
- [Facial Affective Behavior Analysis Method for 5th ABAW Competition](https://mlsanity.com/paper/2303.09145/) (2023)
- [Efficient Feature Extraction and Late Fusion Strategy for Audiovisual Emotional Mimicry Intensity Estimation](https://mlsanity.com/paper/2403.11757/) (2024)
- [Unimodal Multi-Task Fusion for Emotional Mimicry Intensity Prediction](https://mlsanity.com/paper/2403.11879/) (2024)

## Built on by (most cited first, 15 of 15)

- [BAH Dataset for Ambivalence/Hesitancy Recognition in Videos for Digital Behavioural Change](https://mlsanity.com/paper/2505.19328/) (2025)
- [MAVEN: Multi-modal Attention for Valence-Arousal Emotion Network](https://mlsanity.com/paper/2503.12623/) (2025)
- [BROTHER: Behavioral Recognition Optimized Through Heterogeneous Ensemble Regularization for Ambivalence and Hesitancy](https://mlsanity.com/paper/2603.14361/) (2026)
- [Semantic Matters: Multimodal Features for Affective Analysis](https://mlsanity.com/paper/2504.11460/) (2025)
- [Solution for 10th Competition on Ambivalence/Hesitancy (AH) Video Recognition Challenge using Divergence-Based Multimodal Fusion](https://mlsanity.com/paper/2603.16939/) (2026)
- [Multimodal Emotion Regression with Multi-Objective Optimization and VAD-Aware Audio Modeling for the 10th ABAW EMI Track](https://mlsanity.com/paper/2603.13760/) (2026)
- [Solution to the 10th ABAW Expression Recognition Challenge: A Robust Multimodal Framework with Safe Cross-Attention and Modality Dropout](https://mlsanity.com/paper/2603.08034/) (2026)
- [A Two-Stage Dual-Modality Model for Facial Emotional Expression Recognition](https://mlsanity.com/paper/2603.12221/) (2026)
- [Anchoring Emotions in Text: Robust Multimodal Fusion for Mimicry Intensity Estimation](https://mlsanity.com/paper/2603.14976/) (2026)
- [Simple Features and Honest Calibration for Ambivalence and Hesitancy Recognition in Video](https://mlsanity.com/paper/2607.11120/) (2026)
- [HSEmotion Team at the 11th ABAW Challenge: Multi-Task Learning and Ambivalence/Hesitancy Video Recognition](https://mlsanity.com/paper/2607.12774/) (2026)
- [United we stand, Divided we fall: Handling Weak Complementary Relationships for Audio-Visual Emotion Recognition in Valence-Arousal Space](https://mlsanity.com/paper/2503.12261/) (2025)
- [Two-Stage Multimodal Framework for Emotion Mimicry Intensity Prediction](https://mlsanity.com/paper/2605.21869/) (2026)
- [CALM-AH: An ABAW11-Calibrated Multimodal Ensemble with Reliability-Gated Multi-Expert Consensus for Video-Level Ambivalence and Hesitancy Recognition](https://mlsanity.com/paper/2607.29310/) (2026)
- [Modality Discrepancy Transformer for Ambivalence and Hesitancy Recognition](https://mlsanity.com/paper/2609.19148/) (2026)

## Abstract (arXiv)

This article presents our results for the eighth Affective Behavior Analysis in-the-Wild (ABAW) competition. We combine facial emotional descriptors extracted by pre-trained models, namely, our EmotiEffLib library, with acoustic features and embeddings of texts recognized from speech. The frame-level features are aggregated and fed into simple classifiers, e.g., multi-layered perceptron (feed-forward neural network with one hidden layer), to predict ambivalence/hesitancy and facial expressions. In the latter case, we also use the pre-trained facial expression recognition model to select high-score video frames and prevent their processing with a domain-specific video classifier. The video-level prediction of emotional mimicry intensity is implemented by simply aggregating frame-level features and training a multi-layered perceptron. Experimental results for three tasks from the ABAW challenge demonstrate that our approach significantly increases validation metrics compared to existing baselines.