# AVA-ActiveSpeaker: An Audio-Visual Dataset for Active Speaker Detection

arXiv: https://arxiv.org/abs/1901.01342 · HTML: https://mlsanity.com/paper/1901.01342/

## Key facts

- Cited by 83 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Jan 2019.

## Built on by (most cited first, 30 of 83)

- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [Self-Supervised Learning for Videos: A Survey](https://mlsanity.com/paper/2207.00419/) (2022)
- [Sub-word Level Lip Reading With Visual Attention](https://mlsanity.com/paper/2110.07603/) (2021)
- [ACAV100M: Automatic Curation of Large-Scale Datasets for Audio-Visual Video Representation Learning](https://mlsanity.com/paper/2101.10803/) (2021)
- [RenderMe-360: A Large Digital Asset Library and Benchmarks Towards High-fidelity Head Avatars](https://mlsanity.com/paper/2305.13353/) (2023)
- [Learning in Audio-visual Context: A Review, Analysis, and New Perspective](https://mlsanity.com/paper/2208.09579/) (2022)
- [MIntRec: A New Dataset for Multimodal Intent Recognition](https://mlsanity.com/paper/2209.04355/) (2022)
- [A Light Weight Model for Active Speaker Detection](https://mlsanity.com/paper/2303.04439/) (2023)
- [MAAS: Multi-modal Assignation for Active Speaker Detection](https://mlsanity.com/paper/2101.03682/) (2021)
- [The Right to Talk: An Audio-Visual Transformer Approach](https://mlsanity.com/paper/2108.03256/) (2021)
- [Novel-View Acoustic Synthesis](https://mlsanity.com/paper/2301.08730/) (2023)
- [How to Design a Three-Stage Architecture for Audio-Visual Active Speaker Detection in the Wild](https://mlsanity.com/paper/2106.03932/) (2021)
- [Learning Long-Term Spatial-Temporal Graphs for Active Speaker Detection](https://mlsanity.com/paper/2207.07783/) (2022)
- [UniCon: Unified Context Network for Robust Active Speaker Detection](https://mlsanity.com/paper/2108.02607/) (2021)
- [Egocentric Deep Multi-Channel Audio-Visual Active Speaker Localization](https://mlsanity.com/paper/2201.01928/) (2022)
- [AVA-AVD: Audio-Visual Speaker Diarization in the Wild](https://mlsanity.com/paper/2111.14448/) (2021)
- [End-to-End Active Speaker Detection](https://mlsanity.com/paper/2203.14250/) (2022)
- [LoCoNet: Long-Short Context Network for Active Speaker Detection](https://mlsanity.com/paper/2301.08237/) (2023)
- [Look Who's Talking: Active Speaker Detection in the Wild](https://mlsanity.com/paper/2108.07640/) (2021)
- [Look&Listen: Multi-Modal Correlation Learning for Active Speaker Detection and Speech Enhancement](https://mlsanity.com/paper/2203.02216/) (2022)
- [Listen to Look into the Future: Audio-Visual Egocentric Gaze Anticipation](https://mlsanity.com/paper/2305.03907/) (2023)
- [See, Hear, and Understand: Benchmarking Audiovisual Human Speech Understanding in Multimodal Large Language Models](https://mlsanity.com/paper/2512.02231/) (2025)
- [Egocentric Auditory Attention Localization in Conversations](https://mlsanity.com/paper/2303.16024/) (2023)
- [Structure from Silence: Learning Scene Structure from Ambient Sound](https://mlsanity.com/paper/2111.05846/) (2021)
- [TalkNCE: Improving Active Speaker Detection with Talk-Aware Contrastive Learning](https://mlsanity.com/paper/2309.12306/) (2023)
- [AdVerb: Visually Guided Audio Dereverberation](https://mlsanity.com/paper/2308.12370/) (2023)
- [Finding Fallen Objects Via Asynchronous Audio-Visual Integration](https://mlsanity.com/paper/2207.03483/) (2022)
- [Joint Audio-Visual Idling Vehicle Detection with Streamlined Input Dependencies](https://mlsanity.com/paper/2410.21170/) (2024)
- [Rethinking Audio-visual Synchronization for Active Speaker Detection](https://mlsanity.com/paper/2206.10421/) (2022)
- [Intel Labs at Ego4D Challenge 2022: A Better Baseline for Audio-Visual Diarization](https://mlsanity.com/paper/2210.07764/) (2022)

## Abstract (arXiv)

Active speaker detection is an important component in video analysis algorithms for applications such as speaker diarization, video re-targeting for meetings, speech enhancement, and human-robot interaction. The absence of a large, carefully labeled audio-visual dataset for this task has constrained algorithm evaluations with respect to data diversity, environments, and accuracy. This has made comparisons and improvements difficult. In this paper, we present the AVA Active Speaker detection dataset (AVA-ActiveSpeaker) that will be released publicly to facilitate algorithm development and enable comparisons. The dataset contains temporally labeled face tracks in video, where each face instance is labeled as speaking or not, and whether the speech is audible. This dataset contains about 3.65 million human labeled frames or about 38.5 hours of face tracks, and the corresponding audio. We also present a new audio-visual approach for active speaker detection, and analyze its performance, demonstrating both its strength and the contributions of the dataset.