# Look Who's Talking: Active Speaker Detection in the Wild

arXiv: https://arxiv.org/abs/2108.07640 · HTML: https://mlsanity.com/paper/2108.07640/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 42 references that are in the map.
- First posted on arXiv 17 Aug 2021.

## Builds on (most cited first, 24 of 24)

- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [Lip Reading Sentences in the Wild](https://mlsanity.com/paper/1611.05358/) (2016)
- [S3FD: Single Shot Scale-invariant Face Detector](https://mlsanity.com/paper/1708.05237/) (2017)
- [The Conversation: Deep Audio-Visual Speech Enhancement](https://mlsanity.com/paper/1804.04121/) (2018)
- [AVA-ActiveSpeaker: An Audio-Visual Dataset for Active Speaker Detection](https://mlsanity.com/paper/1901.01342/) (2019)
- [Spot the conversation: speaker diarisation in the wild](https://mlsanity.com/paper/2007.01216/) (2020)
- [Perfect match: Improved cross-modal embeddings for audio-visual synchronisation](https://mlsanity.com/paper/1809.08001/) (2018)
- [The Microsoft 2017 Conversational Speech Recognition System](https://mlsanity.com/paper/1708.06073/) (2017)
- [Utterance-level Aggregation For Speaker Recognition In The Wild](https://mlsanity.com/paper/1902.10107/) (2019)
- [Audio-Visual Speech Enhancement Using Multimodal Deep Convolutional Neural Networks](https://mlsanity.com/paper/1709.00944/) (2017)
- [Looking into Your Speech: Learning Cross-modal Affinity for Audio-visual Speech Separation](https://mlsanity.com/paper/2104.02775/) (2021)
- [Naver at ActivityNet Challenge 2019 -- Task B Active Speaker Detection (AVA)](https://mlsanity.com/paper/1906.10555/) (2019)
- [FaceFilter: Audio-visual speech separation using still images](https://mlsanity.com/paper/2005.07074/) (2020)
- [Audio-Visual Speaker Diarization Based on Spatiotemporal Bayesian Fusion](https://mlsanity.com/paper/1603.09725/) (2016)
- [Who said that?: Audio-visual speaker diarisation of real-world meetings](https://mlsanity.com/paper/1906.10042/) (2019)
- [Seeing voices and hearing voices: learning discriminative embeddings using cross-modal self-supervision](https://mlsanity.com/paper/2004.14326/) (2020)
- [Self-supervised learning for audio-visual speaker diarization](https://mlsanity.com/paper/2002.05314/) (2020)
- [Direct Acoustics-to-Word Models for English Conversational Speech Recognition](https://mlsanity.com/paper/1703.07754/) (2017)
- [Putting a Face to the Voice: Fusing Audio and Visual Signals Across a Video to Determine Speakers](https://mlsanity.com/paper/1706.00079/) (2017)

## Built on by (most cited first, 21 of 21)

- [Visual Speech Recognition for Multiple Languages in the Wild](https://mlsanity.com/paper/2202.13084/) (2022)
- [LoCoNet: Long-Short Context Network for Active Speaker Detection](https://mlsanity.com/paper/2301.08237/) (2023)
- [See, Hear, and Understand: Benchmarking Audiovisual Human Speech Understanding in Multimodal Large Language Models](https://mlsanity.com/paper/2512.02231/) (2025)
- [TalkNCE: Improving Active Speaker Detection with Talk-Aware Contrastive Learning](https://mlsanity.com/paper/2309.12306/) (2023)
- [Rethinking Audio-visual Synchronization for Active Speaker Detection](https://mlsanity.com/paper/2206.10421/) (2022)
- [Using Active Speaker Faces for Diarization in TV shows](https://mlsanity.com/paper/2203.15961/) (2022)
- [WASD: A Wilder Active Speaker Detection Dataset](https://mlsanity.com/paper/2303.05321/) (2023)
- [LASER: Lip Landmark Assisted Speaker Detection for Robustness](https://mlsanity.com/paper/2501.11899/) (2025)
- [Revisiting Active Speaker Detection: An In-the-Wild Benchmark for Generalization and Robustness](https://mlsanity.com/paper/2505.21954/) (2025)
- [Spherical World-Locking for Audio-Visual Localization in Egocentric Videos](https://mlsanity.com/paper/2408.05364/) (2024)
- [BIAS: A Body-based Interpretable Active Speaker Approach](https://mlsanity.com/paper/2412.05150/) (2024)
- [ASDnB: Merging Face with Body Cues For Robust Active Speaker Detection](https://mlsanity.com/paper/2412.08594/) (2024)
- [Audio-Visual Activity Guided Cross-Modal Identity Association for Active Speaker Detection](https://mlsanity.com/paper/2212.00539/) (2022)
- [A Multi-Purpose Audio-Visual Corpus for Multi-Modal Persian Speech Recognition: the Arman-AV Dataset](https://mlsanity.com/paper/2301.10180/) (2023)
- [Unsupervised active speaker detection in media content using cross-modal information](https://mlsanity.com/paper/2209.11896/) (2022)
- [GateFusion: Hierarchical Gated Cross-Modal Fusion for Active Speaker Detection](https://mlsanity.com/paper/2512.15707/) (2025)
- [Judge Model for Large-scale Multimodality Benchmarks](https://mlsanity.com/paper/2601.06106/) (2026)
- [Data standardization for robust lip sync](https://mlsanity.com/paper/2202.06198/) (2022)
- [Comparison of Conventional Hybrid and CTC/Attention Decoders for Continuous Visual Speech Recognition](https://mlsanity.com/paper/2402.13004/) (2024)
- [FabuLight-ASD: Unveiling Speech Activity via Body Language](https://mlsanity.com/paper/2411.13674/) (2024)
- [ROAM-ASD: Robust Open-World Active Speaker Detection with Flexible Multimodal Fusion](https://mlsanity.com/paper/2609.26648/) (2026)

## Abstract (arXiv)

In this work, we present a novel audio-visual dataset for active speaker detection in the wild. A speaker is considered active when his or her face is visible and the voice is audible simultaneously. Although active speaker detection is a crucial pre-processing step for many audio-visual tasks, there is no existing dataset of natural human speech to evaluate the performance of active speaker detection. We therefore curate the Active Speakers in the Wild (ASW) dataset which contains videos and co-occurring speech segments with dense speech activity labels. Videos and timestamps of audible segments are parsed and adopted from VoxConverse, an existing speaker diarisation dataset that consists of videos in the wild. Face tracks are extracted from the videos and active segments are annotated based on the timestamps of VoxConverse in a semi-automatic way. Two reference systems, a self-supervised system and a fully supervised one, are evaluated on the dataset to provide the baseline performances of ASW. Cross-domain evaluation is conducted in order to show the negative effect of dubbed videos in the training data.