mlsanityData to 7 Oct 2026

Anchoring Emotions in Text: Robust Multimodal Fusion for Mimicry Intensity Estimation

arXiv 2603.14976 ↗First posted 16 Mar 2026cs.MM cs.CVcited by 1

Not on the map. Why?

Abstract (arXiv)

Estimating Emotional Mimicry Intensity (EMI) in naturalistic environments is a critical yet challenging task in affective computing. The primary difficulty lies in effectively modeling the complex, nonlinear temporal dynamics across highly heterogeneous modalities, especially when physical signals are corrupted or missing. To tackle this, we propose TAEMI (Text-Anchored Emotional Mimicry Intensity estimation), a novel multimodal framework designed for the 10th ABAW Competition. Motivated by the observation that continuous visual and acoustic signals are highly susceptible to transient environmental noise, we break the traditional symmetric fusion paradigm. Instead, we leverage textual transcript--which inherently encode a stable, time-independent semantic prior--as central anchors. Specifically, we introduce a Text-Anchored Dual Cross-Attention mechanism that utilizes these robust textual queries to actively filter out frame-level redundancies and align the noisy physical streams. Furthermore, to prevent catastrophic performance degradation caused by inevitably missing data in unconstrained real-world scenarios, we integrate Learnable Missing-Modality Tokens and a Modality Dropout strategy during training. Extensive experiments on the Hume-Vidmimic2 dataset demonstrate that TAEMI effectively captures fine-grained emotional variations and maintains robust predictive resilience under imperfect conditions. Our framework achieves a state-of-the-art mean Pearson correlation coefficient across six continuous emotional dimensions, significantly outperforming existing baseline methods.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 27 of its 37 references that are in the map. First posted on arXiv 16 Mar 2026.

Builds on

27 of 37 references.

References · 27

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  5. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 637
  6. 2016Temporal Convolutional Networks for Action Segmentation and Detectioncited by 504
  7. 2017Tensor Fusion Network for Multimodal Sentiment Analysiscited by 429
  8. 2020MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysiscited by 262
  9. 2019Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFacecited by 222
  10. 2018Deep Affect Prediction in-the-wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyondcited by 216
  11. 2020Analysing Affective Behavior in the First ABAW 2020 Competitioncited by 178
  12. 2021Affect Analysis in-the-wild: Valence-Arousal, Expressions, Action Units and a Unified Frameworkcited by 174
15 more
  1. 2021Distribution Matching for Heterogeneous Multi-Task Learning: a Large-scale Face Studycited by 168
  2. 2021Analysing Affective Behavior in the second ABAW2 Competitioncited by 163
  3. 2022ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Multi-Task Learning Challengescited by 156
  4. 2019Face Behavior a la carte: Expressions, Affect and Action Units in a Single Networkcited by 155
  5. 2022Dawn of the transformer era in speech emotion recognition: closing the valence gapcited by 155
  6. 2022ABAW: Learning from Synthetic Data & Multi-Task Learning Challengescited by 110
  7. 2023ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Emotional Reaction Intensity Estimation Challengescited by 110
  8. 2024The 6th Affective Behavior Analysis in-the-wild (ABAW) Competitioncited by 78
  9. 2024Distribution Matching for Multi-Task Learning of Classification Tasks: a Large-Scale Study on Faces & Beyondcited by 60
  10. 20247th ABAW Competition: Multi-Task Learning and Compound Expression Recognitioncited by 53
  11. 2024Affective Behaviour Analysis via Integrating Multi-Modal Knowledgecited by 18
  12. 2024Behaviour4All: in-the-wild Facial Behaviour Analysis Toolkitcited by 16
  13. 2025HSEmotion Team at ABAW-8 Competition: Audiovisual Ambivalence/Hesitancy, Emotional Mimicry Intensity and Facial Expression Recognitioncited by 15
  14. 2025DVD: A Comprehensive Dataset for Advancing Violence Detection in Real-World Scenarioscited by 13
  15. 2025Technical Approach for the EMI Challenge in the 8th Affective Behavior Analysis in-the-Wild Competitioncited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.