# Revisiting Active Speaker Detection: An In-the-Wild Benchmark for Generalization and Robustness

arXiv: https://arxiv.org/abs/2505.21954 · HTML: https://mlsanity.com/paper/2505.21954/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 30 references that are in the map.
- First posted on arXiv 28 May 2025.

## Builds on (most cited first, 23 of 23)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [Deep Audio-Visual Speech Recognition](https://mlsanity.com/paper/1809.02108/) (2018)
- [End-to-end Audio-visual Speech Recognition with Conformers](https://mlsanity.com/paper/2102.06657/) (2021)
- [S3FD: Single Shot Scale-invariant Face Detector](https://mlsanity.com/paper/1708.05237/) (2017)
- [Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection](https://mlsanity.com/paper/2107.06592/) (2021)
- [AVA-ActiveSpeaker: An Audio-Visual Dataset for Active Speaker Detection](https://mlsanity.com/paper/1901.01342/) (2019)
- [Spot the conversation: speaker diarisation in the wild](https://mlsanity.com/paper/2007.01216/) (2020)
- [Speaker Diarization with LSTM](https://mlsanity.com/paper/1710.10468/) (2017)
- [A Light Weight Model for Active Speaker Detection](https://mlsanity.com/paper/2303.04439/) (2023)
- [MAAS: Multi-modal Assignation for Active Speaker Detection](https://mlsanity.com/paper/2101.03682/) (2021)
- [How to Design a Three-Stage Architecture for Audio-Visual Active Speaker Detection in the Wild](https://mlsanity.com/paper/2106.03932/) (2021)
- [Learning Long-Term Spatial-Temporal Graphs for Active Speaker Detection](https://mlsanity.com/paper/2207.07783/) (2022)
- [UniCon: Unified Context Network for Robust Active Speaker Detection](https://mlsanity.com/paper/2108.02607/) (2021)
- [LoCoNet: Long-Short Context Network for Active Speaker Detection](https://mlsanity.com/paper/2301.08237/) (2023)
- [Cross-modal Supervision for Learning Active Speaker Detection in Video](https://mlsanity.com/paper/1603.08907/) (2016)
- [Look Who's Talking: Active Speaker Detection in the Wild](https://mlsanity.com/paper/2108.07640/) (2021)
- [Look&Listen: Multi-Modal Correlation Learning for Active Speaker Detection and Speech Enhancement](https://mlsanity.com/paper/2203.02216/) (2022)
- [LSTM based Similarity Measurement with Spectral Clustering for Speaker Diarization](https://mlsanity.com/paper/1907.10393/) (2019)
- [TalkNCE: Improving Active Speaker Detection with Talk-Aware Contrastive Learning](https://mlsanity.com/paper/2309.12306/) (2023)
- [Target Active Speaker Detection with Audio-visual Cues](https://mlsanity.com/paper/2305.12831/) (2023)
- [LASER: Lip Landmark Assisted Speaker Detection for Robustness](https://mlsanity.com/paper/2501.11899/) (2025)

## Built on by (most cited first, 6 of 6)

- [See, Hear, and Understand: Benchmarking Audiovisual Human Speech Understanding in Multimodal Large Language Models](https://mlsanity.com/paper/2512.02231/) (2025)
- [LongCat-Video-Avatar 1.5 Technical Report](https://mlsanity.com/paper/2605.26486/) (2026)
- [GateFusion: Hierarchical Gated Cross-Modal Fusion for Active Speaker Detection](https://mlsanity.com/paper/2512.15707/) (2025)
- [QwenVLConnector: A Fast, Unified Medical VLM Chatbot for Fine-Grained Clinical Perception and Text Generation](https://mlsanity.com/paper/2609.23139/) (2026)
- [ROAM-ASD: Robust Open-World Active Speaker Detection with Flexible Multimodal Fusion](https://mlsanity.com/paper/2609.26648/) (2026)
- [CAST-ASD: Causal Attention over Speakers and Time for Online Active Speaker Detection](https://mlsanity.com/paper/2610.14676/) (2026)

## Abstract (arXiv)

We present UniTalk, a novel dataset emphasizing challenging scenarios to enhance model generalization for the task of active speaker detection (ASD). Previously established benchmarks such as AVA predominantly comprise old movies and thus exhibit significant domain gaps with real-world video. In contrast, UniTalk covers diverse video types reflecting challenging real-world conditions, including underrepresented languages, noisy backgrounds, and crowded scenes, while being on par with AVA in scale. Extensive evaluations reveal that ASD remains unsolved under realistic conditions: state-of-the-art models near-perfect on AVA fail to reach saturation on UniTalk. Conversely, models trained on UniTalk generalize better to modern in-the-wild datasets including Talkies and ASW. UniTalk thus establishes a new benchmark for ASD, providing researchers with a valuable resource for developing and evaluating versatile and resilient models.