# Audio-visual Speaker Recognition with a Cross-modal Discriminative Network

arXiv: https://arxiv.org/abs/2008.03894 · HTML: https://mlsanity.com/paper/2008.03894/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Aug 2020.

## Built on by (most cited first, 24 of 24)

- [AVA-AVD: Audio-Visual Speaker Diarization in the Wild](https://mlsanity.com/paper/2111.14448/) (2021)
- [A Multi-View Approach To Audio-Visual Speaker Verification](https://mlsanity.com/paper/2102.06291/) (2021)
- [Multimodal Action Quality Assessment](https://mlsanity.com/paper/2402.09444/) (2024)
- [Single-branch Network for Multimodal Training](https://mlsanity.com/paper/2303.06129/) (2023)
- [MFA: TDNN with Multi-scale Frequency-channel Attention for Text-independent Speaker Verification with Short Utterances](https://mlsanity.com/paper/2202.01624/) (2022)
- [Cross-Modal Perceptionist: Can Face Geometry be Gleaned from Voices?](https://mlsanity.com/paper/2203.09824/) (2022)
- [Face-voice Association in Multilingual Environments (FAME) 2026 Challenge Evaluation Plan](https://mlsanity.com/paper/2508.04592/) (2025)
- [Linking Faces and Voices Across Languages: Insights from the FAME 2026 Challenge](https://mlsanity.com/paper/2512.20376/) (2025)
- [CN-Celeb-AV: A Multi-Genre Audio-Visual Dataset for Person Recognition](https://mlsanity.com/paper/2305.16049/) (2023)
- [Audio-Visual Speaker Verification via Joint Cross-Attention](https://mlsanity.com/paper/2309.16569/) (2023)
- [Face-voice Association in Multilingual Environments (FAME) Challenge 2024 Evaluation Plan](https://mlsanity.com/paper/2404.09342/) (2024)
- [Learning Branched Fusion and Orthogonal Projection for Face-Voice Association](https://mlsanity.com/paper/2208.10238/) (2022)
- [A Multimodal Sensor Fusion Framework Robust to Missing Modalities for Person Recognition](https://mlsanity.com/paper/2210.10972/) (2022)
- [Cross-modal Audio-visual Co-learning for Text-independent Speaker Verification](https://mlsanity.com/paper/2302.11254/) (2023)
- [Rethinking Voice-Face Correlation: A Geometry View](https://mlsanity.com/paper/2307.13948/) (2023)
- [VoxVietnam: a Large-Scale Multi-Genre Dataset for Vietnamese Speaker Recognition](https://mlsanity.com/paper/2501.00328/) (2024)
- [Comparative Analysis of Modality Fusion Approaches for Audio-Visual Person Identification and Verification](https://mlsanity.com/paper/2409.00562/) (2024)
- [Graph-based Multi-View Fusion and Local Adaptation: Mitigating Within-Household Confusability for Speaker Identification](https://mlsanity.com/paper/2207.04081/) (2022)
- [Getting More for Less: Using Weak Labels and AV-Mixup for Robust Audio-Visual Speaker Verification](https://mlsanity.com/paper/2309.07115/) (2023)
- [POLY-SIM: Polyglot Speaker Identification with Missing Modality Grand Challenge 2026 Evaluation Plan](https://mlsanity.com/paper/2603.24569/) (2026)
- [Exploring Deep Learning for Joint Audio-Visual Lip Biometrics](https://mlsanity.com/paper/2104.08510/) (2021)
- [SSAVSV: Towards Unified Model for Self-Supervised Audio-Visual Speaker Verification](https://mlsanity.com/paper/2506.17694/) (2025)
- [Learning Speaker Identity Beyond Language and Modality Constraints: Insights from the POLY-SIM 2026 Challenge](https://mlsanity.com/paper/2607.13669/) (2026)
- [Face and Voice Cross-modal Association with Learning Convex Feature Embedding](https://mlsanity.com/paper/2607.28129/) (2026)

## Abstract (arXiv)

Audio-visual speaker recognition is one of the tasks in the recent 2019 NIST speaker recognition evaluation (SRE). Studies in neuroscience and computer science all point to the fact that vision and auditory neural signals interact in the cognitive process. This motivated us to study a cross-modal network, namely voice-face discriminative network (VFNet) that establishes the general relation between human voice and face. Experiments show that VFNet provides additional speaker discriminative information. With VFNet, we achieve 16.54% equal error rate relative reduction over the score level fusion audio-visual baseline on evaluation set of 2019 NIST SRE.