# AVA-Speech: A Densely Labeled Dataset of Speech Activity in Movies

arXiv: https://arxiv.org/abs/1808.00606 · HTML: https://mlsanity.com/paper/1808.00606/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Aug 2018.

## Built on by (most cited first, 10 of 10)

- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [ACAV100M: Automatic Curation of Large-Scale Datasets for Audio-Visual Video Representation Learning](https://mlsanity.com/paper/2101.10803/) (2021)
- [SayAnything: Audio-Driven Lip Synchronization with Conditional Video Diffusion](https://mlsanity.com/paper/2502.11515/) (2025)
- [Long-range Multimodal Pretraining for Movie Understanding](https://mlsanity.com/paper/2308.09775/) (2023)
- [ZR-2021VG: Zero-Resource Speech Challenge, Visually-Grounded Language Modelling track, 2021 edition](https://mlsanity.com/paper/2107.06546/) (2021)
- [R^3-VQA: "Read the Room" by Video Social Reasoning](https://mlsanity.com/paper/2505.04147/) (2025)
- [Robust Active Speaker Detection in Noisy Environments](https://mlsanity.com/paper/2403.19002/) (2024)
- [Tiny Noise-Robust Voice Activity Detector for Voice Assistants](https://mlsanity.com/paper/2507.22157/) (2025)
- [NAS-VAD: Neural Architecture Search for Voice Activity Detection](https://mlsanity.com/paper/2201.09032/) (2022)
- [VAD to the Bone: Ultra-Tiny Speech Activity Detection for Edge Deployment](https://mlsanity.com/paper/2607.25870/) (2026)

## Abstract (arXiv)

Speech activity detection (or endpointing) is an important processing step for applications such as speech recognition, language identification and speaker diarization. Both audio- and vision-based approaches have been used for this task in various settings, often tailored toward end applications. However, much of the prior work reports results in synthetic settings, on task-specific datasets, or on datasets that are not openly available. This makes it difficult to compare approaches and understand their strengths and weaknesses. In this paper, we describe a new dataset which we will release publicly containing densely labeled speech activity in YouTube videos, with the goal of creating a shared, available dataset for this task. The labels in the dataset annotate three different speech activity conditions: clean speech, speech co-occurring with music, and speech co-occurring with noise, which enable analysis of model performance in more challenging conditions based on the presence of overlapping noise. We report benchmark performance numbers on AVA-Speech using off-the-shelf, state-of-the-art audio and vision models that serve as a baseline to facilitate future research.