# Active Speakers in Context

arXiv: https://arxiv.org/abs/2005.09812 · HTML: https://mlsanity.com/paper/2005.09812/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 May 2020.

## Built on by (most cited first, 30 of 41)

- [Sub-word Level Lip Reading With Visual Attention](https://mlsanity.com/paper/2110.07603/) (2021)
- [A Light Weight Model for Active Speaker Detection](https://mlsanity.com/paper/2303.04439/) (2023)
- [MAAS: Multi-modal Assignation for Active Speaker Detection](https://mlsanity.com/paper/2101.03682/) (2021)
- [How to Design a Three-Stage Architecture for Audio-Visual Active Speaker Detection in the Wild](https://mlsanity.com/paper/2106.03932/) (2021)
- [Learning Long-Term Spatial-Temporal Graphs for Active Speaker Detection](https://mlsanity.com/paper/2207.07783/) (2022)
- [UniCon: Unified Context Network for Robust Active Speaker Detection](https://mlsanity.com/paper/2108.02607/) (2021)
- [End-to-End Active Speaker Detection](https://mlsanity.com/paper/2203.14250/) (2022)
- [LoCoNet: Long-Short Context Network for Active Speaker Detection](https://mlsanity.com/paper/2301.08237/) (2023)
- [Look&Listen: Multi-Modal Correlation Learning for Active Speaker Detection and Speech Enhancement](https://mlsanity.com/paper/2203.02216/) (2022)
- [Listen to Look into the Future: Audio-Visual Egocentric Gaze Anticipation](https://mlsanity.com/paper/2305.03907/) (2023)
- [Learning to Cut by Watching Movies](https://mlsanity.com/paper/2108.04294/) (2021)
- [Egocentric Auditory Attention Localization in Conversations](https://mlsanity.com/paper/2303.16024/) (2023)
- [TalkNCE: Improving Active Speaker Detection with Talk-Aware Contrastive Learning](https://mlsanity.com/paper/2309.12306/) (2023)
- [AdVerb: Visually Guided Audio Dereverberation](https://mlsanity.com/paper/2308.12370/) (2023)
- [Learning Spatial Features from Audio-Visual Correspondence in Egocentric Videos](https://mlsanity.com/paper/2307.04760/) (2023)
- [Using Active Speaker Faces for Diarization in TV shows](https://mlsanity.com/paper/2203.15961/) (2022)
- [WASD: A Wilder Active Speaker Detection Dataset](https://mlsanity.com/paper/2303.05321/) (2023)
- [LASER: Lip Landmark Assisted Speaker Detection for Robustness](https://mlsanity.com/paper/2501.11899/) (2025)
- [ASDnB: Merging Face with Body Cues For Robust Active Speaker Detection](https://mlsanity.com/paper/2412.08594/) (2024)
- [FaVoA: Face-Voice Association Favours Ambiguous Speaker Detection](https://mlsanity.com/paper/2109.00577/) (2021)
- [Pairwise Emotional Relationship Recognition in Drama Videos: Dataset and Benchmark](https://mlsanity.com/paper/2109.11243/) (2021)
- [Learning Spatial-Temporal Graphs for Active Speaker Detection](https://mlsanity.com/paper/2112.01479/) (2021)
- [Audio-Visual Activity Guided Cross-Modal Identity Association for Active Speaker Detection](https://mlsanity.com/paper/2212.00539/) (2022)
- [Active Speaker Detection as a Multi-Objective Optimization with Uncertainty-based Multimodal Fusion](https://mlsanity.com/paper/2106.03821/) (2021)
- [Push-Pull: Characterizing the Adversarial Robustness for Audio-Visual Active Speaker Detection](https://mlsanity.com/paper/2210.00753/) (2022)
- [Whose Emotion Matters? Speaking Activity Localisation without Prior Knowledge](https://mlsanity.com/paper/2211.15377/) (2022)
- [UniCon+: ICTCAS-UCAS Submission to the AVA-ActiveSpeaker Task at ActivityNet Challenge 2022](https://mlsanity.com/paper/2206.10861/) (2022)
- [Robust Active Speaker Detection in Noisy Environments](https://mlsanity.com/paper/2403.19002/) (2024)
- [Joint Learning of Visual-Audio Saliency Prediction and Sound Source Localization on Multi-face Videos](https://mlsanity.com/paper/2111.08567/) (2021)
- [End-To-End Audiovisual Feature Fusion for Active Speaker Detection](https://mlsanity.com/paper/2207.13434/) (2022)

## Abstract (arXiv)

Current methods for active speak er detection focus on modeling short-term audiovisual information from a single speaker. Although this strategy can be enough for addressing single-speaker scenarios, it prevents accurate detection when the task is to identify who of many candidate speakers are talking. This paper introduces the Active Speaker Context, a novel representation that models relationships between multiple speakers over long time horizons. Our Active Speaker Context is designed to learn pairwise and temporal relations from an structured ensemble of audio-visual observations. Our experiments show that a structured feature ensemble already benefits the active speaker detection performance. Moreover, we find that the proposed Active Speaker Context improves the state-of-the-art on the AVA-ActiveSpeaker dataset achieving a mAP of 87.1%. We present ablation studies that verify that this result is a direct consequence of our long-term multi-speaker analysis.