# Cross-modal Supervision for Learning Active Speaker Detection in Video

arXiv: https://arxiv.org/abs/1603.08907 · HTML: https://mlsanity.com/paper/1603.08907/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Mar 2016.

## Built on by (most cited first, 21 of 21)

- [A Light Weight Model for Active Speaker Detection](https://mlsanity.com/paper/2303.04439/) (2023)
- [The Right to Talk: An Audio-Visual Transformer Approach](https://mlsanity.com/paper/2108.03256/) (2021)
- [Learning Long-Term Spatial-Temporal Graphs for Active Speaker Detection](https://mlsanity.com/paper/2207.07783/) (2022)
- [UniCon: Unified Context Network for Robust Active Speaker Detection](https://mlsanity.com/paper/2108.02607/) (2021)
- [LoCoNet: Long-Short Context Network for Active Speaker Detection](https://mlsanity.com/paper/2301.08237/) (2023)
- [Look&Listen: Multi-Modal Correlation Learning for Active Speaker Detection and Speech Enhancement](https://mlsanity.com/paper/2203.02216/) (2022)
- [WASD: A Wilder Active Speaker Detection Dataset](https://mlsanity.com/paper/2303.05321/) (2023)
- [Revisiting Active Speaker Detection: An In-the-Wild Benchmark for Generalization and Robustness](https://mlsanity.com/paper/2505.21954/) (2025)
- [ASDnB: Merging Face with Body Cues For Robust Active Speaker Detection](https://mlsanity.com/paper/2412.08594/) (2024)
- [Learning Spatial-Temporal Graphs for Active Speaker Detection](https://mlsanity.com/paper/2112.01479/) (2021)
- [Audio-Visual Activity Guided Cross-Modal Identity Association for Active Speaker Detection](https://mlsanity.com/paper/2212.00539/) (2022)
- [Active Speaker Detection as a Multi-Objective Optimization with Uncertainty-based Multimodal Fusion](https://mlsanity.com/paper/2106.03821/) (2021)
- [HumanOmni-Speaker: Identifying Who said What and When](https://mlsanity.com/paper/2603.21664/) (2026)
- [Joint Learning of Visual-Audio Saliency Prediction and Sound Source Localization on Multi-face Videos](https://mlsanity.com/paper/2111.08567/) (2021)
- [End-To-End Audiovisual Feature Fusion for Active Speaker Detection](https://mlsanity.com/paper/2207.13434/) (2022)
- [Unsupervised active speaker detection in media content using cross-modal information](https://mlsanity.com/paper/2209.11896/) (2022)
- [REWIND Dataset: Privacy-preserving Speaking Status Segmentation from Multimodal Body Movement Signals in the Wild](https://mlsanity.com/paper/2403.01229/) (2024)
- [Imitation of human motion achieves natural head movements for humanoid robots in an active-speaker detection task](https://mlsanity.com/paper/2407.11915/) (2024)
- [An Efficient and Streaming Audio Visual Active Speaker Detection System](https://mlsanity.com/paper/2409.09018/) (2024)
- [CLIP-VAD: Exploiting Vision-Language Models for Voice Activity Detection](https://mlsanity.com/paper/2410.14509/) (2024)
- [C3ASD: Multi-Level Consistency-Driven Representation Learning for Robust Active Speaker Detection](https://mlsanity.com/paper/2607.03018/) (2026)

## Abstract (arXiv)

In this paper, we show how to use audio to supervise the learning of active speaker detection in video. Voice Activity Detection (VAD) guides the learning of the vision-based classifier in a weakly supervised manner. The classifier uses spatio-temporal features to encode upper body motion - facial expressions and gesticulations associated with speaking. We further improve a generic model for active speaker detection by learning person specific models. Finally, we demonstrate the online adaptation of generic models learnt on one dataset, to previously unseen people in a new dataset, again using audio (VAD) for weak supervision. The use of temporal continuity overcomes the lack of clean training data. We are the first to present an active speaker detection system that learns on one audio-visual dataset and automatically adapts to speakers in a new dataset. This work can be seen as an example of how the availability of multi-modal data allows us to learn a model without the need for supervision, by transferring knowledge from one modality to another.