# Adaptive Multimodal Person Recognition: A Robust Framework for Handling Missing Modalities

arXiv: https://arxiv.org/abs/2512.14961 · HTML: https://mlsanity.com/paper/2512.14961/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 36 references that are in the map.
- First posted on arXiv 16 Dec 2025.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [MagFace: A Universal Representation for Face Recognition and Quality Assessment](https://mlsanity.com/paper/2103.06627/) (2021)
- [Uncertainty Modeling for Out-of-Distribution Generalization](https://mlsanity.com/paper/2202.03958/) (2022)
- [Seeing Voices and Hearing Faces: Cross-modal biometric matching](https://mlsanity.com/paper/1804.00326/) (2018)
- [VoxBlink2: A 100K+ Speaker Recognition Corpus and the Open-Set Speaker-Identification Benchmark](https://mlsanity.com/paper/2407.11510/) (2024)
- [Masked Modeling Duo: Towards a Universal Audio Pre-training Framework](https://mlsanity.com/paper/2404.06095/) (2024)
- [All in One Framework for Multimodal Re-identification in the Wild](https://mlsanity.com/paper/2405.04741/) (2024)
- [Delving into VoxCeleb: environment invariant speaker recognition](https://mlsanity.com/paper/1910.11238/) (2019)
- [Speaker Recognition in Realistic Scenario Using Multimodal Data](https://mlsanity.com/paper/2302.13033/) (2023)
- [SAIC: Integration of Speech Anonymization and Identity Classification](https://mlsanity.com/paper/2312.15190/) (2023)
- [From Face to Gait: Weakly-Supervised Learning of Gender Information from Walking Patterns](https://mlsanity.com/paper/2111.00538/) (2021)
- [Facial Emotion Recognition Under Mask Coverage Using a Data Augmentation Technique](https://mlsanity.com/paper/2312.01335/) (2023)
- [Audio-Visual Person Verification based on Recursive Fusion of Joint Cross-Attention](https://mlsanity.com/paper/2403.04654/) (2024)
- [Comparative Analysis of Modality Fusion Approaches for Audio-Visual Person Identification and Verification](https://mlsanity.com/paper/2409.00562/) (2024)
- [Towards Language-Independent Face-Voice Association with Multimodal Foundation Models](https://mlsanity.com/paper/2512.02759/) (2025)

## Abstract (arXiv)

Person identification systems often rely on audio, visual, or behavioral cues, but real-world conditions frequently present with missing or degraded modalities. To address this challenge, we propose a multimodal person identification framework incorporating upper-body motion, face, and voice. Experimental results demonstrate that body motion outperforms traditional modalities such as face and voice in within-session evaluations, while serving as a complementary cue that enhances performance in multi-session scenarios. Our model employs a unified hybrid fusion strategy, fusing both feature-level and score-level information to maximize representational richness and decision accuracy. Specifically, it leverages multi-task learning to process modalities independently, followed by cross-attention and gated fusion mechanisms to exploit both unimodal information and cross-modal interactions. Finally, a confidence-weighted strategy and mistake-correction mechanism dynamically adapt to missing data, ensuring that our single classification head achieves optimal performance even in unimodal and bimodal scenarios. We evaluate our method on CANDOR, a newly introduced interview-based multimodal dataset, which we benchmark in this work for the first time. Our results demonstrate that the proposed trimodal system achieves 99.51% Top-1 accuracy on person identification tasks. In addition, we evaluate our model on the VoxCeleb1 dataset as a widely used evaluation protocol and reach 99.92% accuracy in bimodal mode, outperforming conventional approaches. Moreover, we show that our system maintains high accuracy even when one or two modalities are unavailable, making it a robust solution for real-world person recognition applications. The code and data for this work are publicly available.