# Learning Visual Voice Activity Detection with an Automatically Annotated Dataset

arXiv: https://arxiv.org/abs/2009.11204 · HTML: https://mlsanity.com/paper/2009.11204/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Sep 2020.

## Built on by (most cited first, 3 of 3)

- [Sub-word Level Lip Reading With Visual Attention](https://mlsanity.com/paper/2110.07603/) (2021)
- [A Real-Time Active Speaker Detection System Integrating an Audio-Visual Signal with a Spatial Querying Mechanism](https://mlsanity.com/paper/2309.08295/) (2023)
- [The VVAD-LRS3 Dataset for Visual Voice Activity Detection](https://mlsanity.com/paper/2109.13789/) (2021)

## Abstract (arXiv)

Visual voice activity detection (V-VAD) uses visual features to predict whether a person is speaking or not. V-VAD is useful whenever audio VAD (A-VAD) is inefficient either because the acoustic signal is difficult to analyze or because it is simply missing. We propose two deep architectures for V-VAD, one based on facial landmarks and one based on optical flow. Moreover, available datasets, used for learning and for testing V-VAD, lack content variability. We introduce a novel methodology to automatically create and annotate very large datasets in-the-wild -- WildVVAD -- based on combining A-VAD with face detection and tracking. A thorough empirical evaluation shows the advantage of training the proposed deep V-VAD models with this dataset.