mlsanityData to 7 Oct 2026

Quality-Aware Multimodal Fusion Reveals Implicit Identity in Valence-Arousal Features

arXiv 2607.21347 ↗First posted 23 Jul 2026cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Conventional face recognition relies on static appearance cues and degrades in unconstrained settings with expression variation, occlusion, and poor lighting. We hypothesize that audiovisual expression dynamics carry identity-discriminative information complementary to static appearance, and that extracting this signal requires multimodal representations robust to the variable input quality of in-the-wild video. To learn such representations, we cast multimodal valence-arousal (VA) estimation as a pretext task and propose Quality-Aware Adaptive Fusion (QAAF), which estimates per-sample, per-modality reliability and adapts each modality's contribution through learned soft gating and a quality-dependent dropout. For the problem of VA estimation, QAAF achieves an average Concordance Correlation Coefficient (CCC) of 0.472 via late fusion ensembling on Aff-wild2, improving over a baseline ensemble under the same setting (0.415) as well as a single-backbone baseline (0.288). Furthermore, the proposed QAAF demonstrates greater resilience to unavailable modalities, with only a 7.5-34.4% relative decrease in CCC when one modality is missing. We then probe whether these VA-trained features encode identity without identity-specific training. On AFEW-VA (67 actors) and YTF (1,595 subjects), VA-trained backbone features rank first among evaluated soft biometric methods, and score-level fusion with ArcFace lowers EER on both datasets (0.022 to 0.021 on AFEW-VA, 0.106 to 0.104 on YTF), correcting 68.2% of ArcFace's false accepts on AFEW-VA. These findings establish multimodal VA estimation as a soft biometric modality complementary to conventional face recognition.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 31 of its 50 references that are in the map. First posted on arXiv 23 Jul 2026.

Builds on

31 of 50 references.

References · 31

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  3. 2021ViViT: A Video Vision Transformercited by 1,610
  4. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  5. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
  6. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 637
  7. 2022AdaFace: Quality Adaptive Margin for Face Recognitioncited by 390
  8. 2022Balanced Multimodal Learning via On-the-fly Gradient Modulationcited by 259
  9. 2021SMIL: Multimodal Learning with Severely Missing Modalitycited by 243
  10. 2018Deep Affect Prediction in-the-wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyondcited by 216
  11. 2023Provable Dynamic Fusion for Low-Quality Multimodal Datacited by 81
  12. 2023Multimodal Representation Learning by Alternating Unimodal Adaptationcited by 74
19 more
  1. 2017End-to-End Multimodal Emotion Recognition using Deep Neural Networkscited by 69
  2. 2021Distract Your Attention: Multi-head Cross Attention Network for Facial Expression Recognitioncited by 55
  3. 2023EdgeFace: Efficient Face Recognition Model for Edge Devicescited by 33
  4. 2023Robust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptationcited by 27
  5. 2024Multimodal Prompt Learning with Missing Modalities for Sentiment Analysis and Emotion Recognitioncited by 26
  6. 2023DifFIQA: Face Image Quality Assessment Using Denoising Diffusion Probabilistic Modelscited by 22
  7. 2022SFace: Sigmoid-Constrained Hypersphere Loss for Robust Face Recognitioncited by 21
  8. 2024Predictive Dynamic Fusioncited by 17
  9. 2024Recursive Joint Cross-Modal Attention for Multimodal Fusion in Dimensional Emotion Recognitioncited by 15
  10. 2024Joint Multimodal Transformer for Emotion Recognition in the Wildcited by 13
  11. 2024A Study of Dropout-Induced Modality Bias on Robustness to Missing Video Frames for Audio-Visual Speech Recognitioncited by 12
  12. 2024MMP: Towards Robust Multi-Modal Learning with Masked Modality Projectioncited by 10
  13. 2023SynthDistill: Face Recognition with Knowledge Distillation from Synthetic Datacited by 9
  14. 2025MAVEN: Multi-modal Attention for Valence-Arousal Emotion Networkcited by 8
  15. 2023CoNAN: Conditional Neural Aggregation Network For Unconstrained Face Feature Fusioncited by 6
  16. 2025Learning Contrastive Multimodal Fusion with Improved Modality Dropout for Disease Detection and Predictioncited by 5
  17. 2023Weakly Supervised Face and Whole Body Recognition in Turbulent Environmentscited by 4
  18. 2024Dynamic Cross Attention for Audio-Visual Person Verificationcited by 3
  19. 2025Beyond Simple Fusion: Adaptive Gated Fusion for Robust Multimodal Sentiment Analysiscited by 3

Built on by

Not cited yet.