mlsanityData to 14 Oct 2026

How to Design a Three-Stage Architecture for Audio-Visual Active Speaker Detection in the Wild

arXiv 2106.03932 ↗First posted 7 Jun 2021cs.CV cs.LG cs.SDcited by 31

Not on the map. Why?

Abstract (arXiv)

Successful active speaker detection requires a three-stage pipeline: (i) audio-visual encoding for all speakers in the clip, (ii) inter-speaker relation modeling between a reference speaker and the background speakers within each frame, and (iii) temporal modeling for the reference speaker. Each stage of this pipeline plays an important role for the final performance of the created architecture. Based on a series of controlled experiments, this work presents several practical guidelines for audio-visual active speaker detection. Correspondingly, we present a new architecture called ASDNet, which achieves a new state-of-the-art on the AVA-ActiveSpeaker dataset with a mAP of 93.5% outperforming the second best with a large margin of 4.7%. Our code and pretrained models are publicly available.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Builds on 33 of its 55 references that are in the map. First posted on arXiv 7 Jun 2021.

Builds on

33 of 55 references.

References · 33

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  3. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  4. 2017Non-local Neural Networkscited by 2,293
  5. 2018SlowFast Networks for Video Recognitioncited by 1,867
  6. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,304
  7. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,143
  8. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,068
  9. 2018VoxCeleb2: Deep Speaker Recognitioncited by 875
  10. 2017Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?cited by 551
  11. 2015Empirical Evaluation of Rectified Activations in Convolutional Networkcited by 519
  12. 2017Learning Spatio-Temporal Representation with Pseudo-3D Residual Networkscited by 387
21 more
  1. 2018Moments in Time Dataset: one million videos for event understandingcited by 215
  2. 2015Deep clustering: Discriminative embeddings for segmentation and separationcited by 212
  3. 2018Speaker Recognition from Raw Waveform with SincNetcited by 166
  4. 2019Listen to Look: Action Recognition by Previewing Audiocited by 163
  5. 2018Wave-U-Net: A Multi-Scale Neural Network for End-to-End Audio Source Separationcited by 125
  6. 2019AVA-ActiveSpeaker: An Audio-Visual Dataset for Active Speaker Detectioncited by 83
  7. 2018VoiceFilter: Targeted Voice Separation by Speaker-Conditioned Spectrogram Maskingcited by 73
  8. 2017Simple Recurrent Units for Highly Parallelizable Recurrencecited by 61
  9. 2019Resource Efficient 3D Convolutional Neural Networkscited by 50
  10. 2020Active Speakers in Contextcited by 41
  11. 2021MAAS: Multi-modal Assignation for Active Speaker Detectioncited by 36
  12. 2019Naver at ActivityNet Challenge 2019 -- Task B Active Speaker Detection (AVA)cited by 26
  13. 2020Speech2Action: Cross-modal Supervision for Action Recognitioncited by 26
  14. 2017Learning Filterbanks from Raw Speech for Phone Recognitioncited by 25
  15. 2016Audio-Visual Speaker Diarization Based on Spatiotemporal Bayesian Fusioncited by 23
  16. 2019Small-Footprint Keyword Spotting on Raw Audio Data with Sinc-Convolutionscited by 20
  17. 2018Motion Fused Frames: Data Level Fusion Strategy for Hand Gesture Recognitioncited by 15
  18. 2017Raw Waveform-based Audio Classification Using Sample-level CNN Architecturescited by 12
  19. 2019Multichannel Speech Enhancement by Raw Waveform-mapping using Fully Convolutional Networkscited by 4
  20. 2020Lightweight End-to-End Speech Recognition from Raw Audio Data Using Sinc-Convolutionscited by 3
  21. 2019Comparative Analysis of CNN-based Spatiotemporal Reasoning in Videoscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 8; 2023: 8; 2024: 4; 2025: 4; 2026: 5.

31 papers.

Sort:

By area:

  1. 2023Self-Supervised Video Forensics by Audio-Visual Anomaly Detectioncited by 85
  2. 2023A Light Weight Model for Active Speaker Detectioncited by 40
  3. 2022Learning Long-Term Spatial-Temporal Graphs for Active Speaker Detectioncited by 30
  4. 2022Egocentric Deep Multi-Channel Audio-Visual Active Speaker Localizationcited by 27
  5. 2022End-to-End Active Speaker Detectioncited by 25
  6. 2023LoCoNet: Long-Short Context Network for Active Speaker Detectioncited by 25
  7. 2022Look&Listen: Multi-Modal Correlation Learning for Active Speaker Detection and Speech Enhancementcited by 20
  8. 2023Listen to Look into the Future: Audio-Visual Egocentric Gaze Anticipationcited by 20
  9. 2023Egocentric Auditory Attention Localization in Conversationscited by 18
  10. 2023TalkNCE: Improving Active Speaker Detection with Talk-Aware Contrastive Learningcited by 16
  11. 2022Rethinking Audio-visual Synchronization for Active Speaker Detectioncited by 9
  12. 2023Learning Spatial Features from Audio-Visual Correspondence in Egocentric Videoscited by 8
19 more
  1. 2023WASD: A Wilder Active Speaker Detection Datasetcited by 6
  2. 2025LASER: Lip Landmark Assisted Speaker Detection for Robustnesscited by 6
  3. 2025Revisiting Active Speaker Detection: An In-the-Wild Benchmark for Generalization and Robustnesscited by 6
  4. 2024ASDnB: Merging Face with Body Cues For Robust Active Speaker Detectioncited by 5
  5. 2021Learning Spatial-Temporal Graphs for Active Speaker Detectioncited by 4
  6. 2022Push-Pull: Characterizing the Adversarial Robustness for Audio-Visual Active Speaker Detectioncited by 3
  7. 2022UniCon+: ICTCAS-UCAS Submission to the AVA-ActiveSpeaker Task at ActivityNet Challenge 2022cited by 2
  8. 2024Robust Active Speaker Detection in Noisy Environmentscited by 2
  9. 2026MuVAP: Multimodal Multiparty Voice Activity Projection for Turn-taking Prediction in the Wildcited by 2
  10. 2021A trained humanoid robot can perform human-like crossmodal social attention and conflict resolutioncited by 1
  11. 2025GateFusion: Hierarchical Gated Cross-Modal Fusion for Active Speaker Detectioncited by 1
  12. 2022Data standardization for robust lip synccited by 0
  13. 2024CLIP-VAD: Exploiting Vision-Language Models for Voice Activity Detectioncited by 0
  14. 2024FabuLight-ASD: Unveiling Speech Activity via Body Languagecited by 0
  15. 2025Bitbox: Behavioral Imaging Toolbox for Computational Analysis of Behavior from Videoscited by 0
  16. 2026C3ASD: Multi-Level Consistency-Driven Representation Learning for Robust Active Speaker Detectioncited by 0
  17. 2026Talking to Me or Someone Else? Rethinking Talk-to-Me Detection in Egocentric Videoscited by 0
  18. 2026ROAM-ASD: Robust Open-World Active Speaker Detection with Flexible Multimodal Fusioncited by 0
  19. 2026CAST-ASD: Causal Attention over Speakers and Time for Online Active Speaker Detectioncited by 0