# AMR: Adaptive Modality Routing for Multimodal Polyglot Speaker Identification

arXiv: https://arxiv.org/abs/2606.29335 · HTML: https://mlsanity.com/paper/2606.29335/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 37 references that are in the map.
- First posted on arXiv 28 Jun 2026.

## Builds on (most cited first, 24 of 24)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020)
- [AdapterHub: A Framework for Adapting Transformers](https://mlsanity.com/paper/2007.07779/) (2020)
- [MUSAN: A Music, Speech, and Noise Corpus](https://mlsanity.com/paper/1510.08484/) (2015)
- [AdaFace: Quality Adaptive Margin for Face Recognition](https://mlsanity.com/paper/2204.00964/) (2022)
- [W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training](https://mlsanity.com/paper/2108.06209/) (2021)
- [What Makes Training Multi-Modal Classification Networks Hard?](https://mlsanity.com/paper/1905.12681/) (2019)
- [MagFace: A Universal Representation for Face Recognition and Quality Assessment](https://mlsanity.com/paper/2103.06627/) (2021)
- [WebFace260M: A Benchmark Unveiling the Power of Million-Scale Deep Face Recognition](https://mlsanity.com/paper/2103.04098/) (2021)
- [pyannote.audio: neural building blocks for speaker diarization](https://mlsanity.com/paper/1911.01255/) (2019)
- [CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-training](https://mlsanity.com/paper/2505.17589/) (2025)
- [VoxCPM: Tokenizer-Free TTS for Context-Aware Speech Generation and True-to-Life Voice Cloning](https://mlsanity.com/paper/2509.24650/) (2025)
- [Fusion and Orthogonal Projection for Improved Face-Voice Association](https://mlsanity.com/paper/2112.10483/) (2021)
- [Cross-modal Speaker Verification and Recognition: A Multilingual Perspective](https://mlsanity.com/paper/2004.13780/) (2020)
- [VoxBlink2: A 100K+ Speaker Recognition Corpus and the Open-Set Speaker-Identification Benchmark](https://mlsanity.com/paper/2407.11510/) (2024)
- [Deep Latent Space Learning for Cross-modal Mapping of Audio and Visual Signals](https://mlsanity.com/paper/1909.08685/) (2019)
- [The SpeakIn System for VoxCeleb Speaker Recognition Challange 2021](https://mlsanity.com/paper/2109.01989/) (2021)
- [Single-branch Network for Multimodal Training](https://mlsanity.com/paper/2303.06129/) (2023)
- [PAEFF: Precise Alignment and Enhanced Gated Feature Fusion for Face-Voice Association](https://mlsanity.com/paper/2505.17002/) (2025)
- [Linking Faces and Voices Across Languages: Insights from the FAME 2026 Challenge](https://mlsanity.com/paper/2512.20376/) (2025)

## Built on by (most cited first, 1 of 1)

- [Learning Speaker Identity Beyond Language and Modality Constraints: Insights from the POLY-SIM 2026 Challenge](https://mlsanity.com/paper/2607.13669/) (2026)

## Abstract (arXiv)

Multimodal speaker identification systems face two key challenges in real-world deployment: missing modalities and language mismatch between training and testing conditions. In practical scenarios, background multi-speaker conversations, ambient noise, and overlapping speech further degrade identification accuracy. To address these challenges, we propose a multimodal polyglot speaker identification system for the POLY-SIM 2026 Grand Challenge. The system is fundamentally built upon Adaptive Modality Routing(AMR), a modality fusion module that dynamically assesses per-sample input quality and integrates modality information. Specifically, AMR employs two modality adapters to process the embeddings extracted from a linguistically robust audio encoder(W2V-BERT 2.0) and a large-scale pretrained face encoder(IResNet-18), producing modality-adapted embeddings. Based on these adapted embeddings, a trainable router estimates dynamic modality weights, which are subsequently applied to aggregate the modality-specific logits for the final prediction. To optimize this routing mechanism, we adopt a modality-aware training strategy that constructs four types of sample pairs to simulate diverse input conditions, with KL divergence serving as explicit supervision for weight assignment. Experimental results on the POLY-SIM 2026 evaluation set show that the proposed system achieves identification accuracy of 99.93%(English multimodal, P3), 100.00%(Urdu multimodal, P5), 97.50%(English audio-only, P4), and 98.83%(Urdu audio-only, P6). The average accuracy across all four protocols is 99.07%, surpassing the Fusion and Orthogonal Projection(FOP) baseline by 32.73%.