mlsanityData to 14 Oct 2026

MAAS: Multi-modal Assignation for Active Speaker Detection

arXiv 2101.03682 ↗First posted 11 Jan 2021cs.CVcited by 36

Not on the map. Why?

Abstract (arXiv)

Active speaker detection requires a solid integration of multi-modal cues. While individual modalities can approximate a solution, accurate predictions can only be achieved by explicitly fusing the audio and visual features and modeling their temporal progression. Despite its inherent muti-modal nature, current methods still focus on modeling and fusing short-term audiovisual features for individual speakers, often at frame level. In this paper we present a novel approach to active speaker detection that directly addresses the multi-modal nature of the problem, and provides a straightforward strategy where independent visual features from potential speakers in the scene are assigned to a previously detected speech event. Our experiments show that, an small graph data structure built from a single frame, allows to approximate an instantaneous audio-visual assignment problem. Moreover, the temporal extension of this initial graph achieves a new state-of-the-art on the AVA-ActiveSpeaker dataset with a mAP of 88.8%.

Key facts. Cited by 36 papers in the map (data through 14 Oct 2026). Builds on 38 of its 63 references that are in the map. First posted on arXiv 11 Jan 2021.

Builds on

38 of 63 references.

References · 38

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  3. 2019Fast Graph Representation Learning with PyTorch Geometriccited by 2,434
  4. 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,223
  5. 2016Identity Mappings in Deep Residual Networkscited by 1,965
  6. 2018Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognitioncited by 1,209
  7. 2017Simple Online and Realtime Tracking with a Deep Association Metriccited by 910
  8. 2018VoxCeleb2: Deep Speaker Recognitioncited by 875
  9. 2017VoxCeleb: a large-scale speaker identification datasetcited by 732
  10. 2019DeepGCNs: Can GCNs Go as Deep as CNNs?cited by 527
  11. 2017Scene Graph Generation by Iterative Message Passingcited by 462
  12. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
26 more
  1. 2018Image Generation from Scene Graphscited by 324
  2. 2018Graph R-CNN for Scene Graph Generationcited by 298
  3. 2017Generalized End-to-End Loss for Speaker Verificationcited by 293
  4. 2020DeeperGCN: All You Need to Train Deeper GCNscited by 250
  5. 2019G-TAD: Sub-Graph Localization for Temporal Action Detectioncited by 249
  6. 2015Structural-RNN: Deep Learning on Spatio-Temporal Graphscited by 234
  7. 2016Lip Reading Sentences in the Wildcited by 228
  8. 2020Self-Supervised Learning of Audio-Visual Objects from Videocited by 188
  9. 2018Speaker Recognition from Raw Waveform with SincNetcited by 166
  10. 2019RetinaFace: Single-stage Dense Face Localisation in the Wildcited by 165
  11. 2019AVA-ActiveSpeaker: An Audio-Visual Dataset for Active Speaker Detectioncited by 83
  12. 2020Spot the conversation: speaker diarisation in the wildcited by 79
  13. 2018Factorizable Net: An Efficient Subgraph-based Framework for Scene Graph Generationcited by 78
  14. 2017Speaker Diarization with LSTMcited by 73
  15. 2018VoiceFilter: Targeted Voice Separation by Speaker-Conditioned Spectrogram Maskingcited by 73
  16. 2019DeepGCNs: Making GCNs Go as Deep as CNNscited by 72
  17. 2019SGAS: Sequential Greedy Architecture Searchcited by 69
  18. 2018Seeing Voices and Hearing Faces: Cross-modal biometric matchingcited by 60
  19. 2018Learnable PINs: Cross-Modal Embeddings for Person Identitycited by 60
  20. 2018Perfect match: Improved cross-modal embeddings for audio-visual synchronisationcited by 60
  21. 2018Fully Supervised Speaker Diarizationcited by 47
  22. 2020Active Speakers in Contextcited by 41
  23. 2018On Learning Associations of Faces and Voicescited by 26
  24. 2019Naver at ActivityNet Challenge 2019 -- Task B Active Speaker Detection (AVA)cited by 26
  25. 2019Point Clouds Learning with Attention-based Graph Convolution Networkscited by 8
  26. 2018Neural Predictive Coding using Convolutional Neural Networks towards Unsupervised Learning of Speaker Characteristicscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 3; 2022: 10; 2023: 10; 2024: 3; 2025: 5; 2026: 5.

36 papers.

Sort:

  1. 2023Self-Supervised Video Forensics by Audio-Visual Anomaly Detectioncited by 85
  2. 2021Sub-word Level Lip Reading With Visual Attentioncited by 63
  3. 2022MIntRec: A New Dataset for Multimodal Intent Recognitioncited by 41
  4. 2023A Light Weight Model for Active Speaker Detectioncited by 40
  5. 2021How to Design a Three-Stage Architecture for Audio-Visual Active Speaker Detection in the Wildcited by 31
  6. 2022Learning Long-Term Spatial-Temporal Graphs for Active Speaker Detectioncited by 30
  7. 2022End-to-End Active Speaker Detectioncited by 25
  8. 2023LoCoNet: Long-Short Context Network for Active Speaker Detectioncited by 25
  9. 2022Look&Listen: Multi-Modal Correlation Learning for Active Speaker Detection and Speech Enhancementcited by 20
  10. 2023Listen to Look into the Future: Audio-Visual Egocentric Gaze Anticipationcited by 20
  11. 2023Sound Localization from Motion: Jointly Learning Sound Direction and Camera Rotationcited by 18
  12. 2023Egocentric Auditory Attention Localization in Conversationscited by 18
24 more
  1. 2023TalkNCE: Improving Active Speaker Detection with Talk-Aware Contrastive Learningcited by 16
  2. 2023Learning Spatial Features from Audio-Visual Correspondence in Egocentric Videoscited by 8
  3. 2023WASD: A Wilder Active Speaker Detection Datasetcited by 6
  4. 2025LASER: Lip Landmark Assisted Speaker Detection for Robustnesscited by 6
  5. 2025Revisiting Active Speaker Detection: An In-the-Wild Benchmark for Generalization and Robustnesscited by 6
  6. 2024ASDnB: Merging Face with Body Cues For Robust Active Speaker Detectioncited by 5
  7. 2021Learning Spatial-Temporal Graphs for Active Speaker Detectioncited by 4
  8. 2022OWL (Observe, Watch, Listen): Audiovisual Temporal Context for Localizing Actions in Egocentric Videoscited by 4
  9. 2022Audio-Visual Activity Guided Cross-Modal Identity Association for Active Speaker Detectioncited by 4
  10. 2022Push-Pull: Characterizing the Adversarial Robustness for Audio-Visual Active Speaker Detectioncited by 3
  11. 2022Whose Emotion Matters? Speaking Activity Localisation without Prior Knowledgecited by 3
  12. 2022UniCon+: ICTCAS-UCAS Submission to the AVA-ActiveSpeaker Task at ActivityNet Challenge 2022cited by 2
  13. 2024Robust Active Speaker Detection in Noisy Environmentscited by 2
  14. 2022Unsupervised active speaker detection in media content using cross-modal informationcited by 1
  15. 2023Multi-Resolution Audio-Visual Feature Fusion for Temporal Action Localizationcited by 1
  16. 2025CoGenAV: Versatile Audio-Visual Representation Learning via Contrastive-Generative Synchronizationcited by 1
  17. 2025DEL: Dense Event Localization for Multi-modal Audio-Visual Understandingcited by 1
  18. 2025GateFusion: Hierarchical Gated Cross-Modal Fusion for Active Speaker Detectioncited by 1
  19. 2024FabuLight-ASD: Unveiling Speech Activity via Body Languagecited by 0
  20. 2026Towards Accurate and Robust Surveillance Roadside IVD via Trackletized Audio-Visual Reasoningcited by 0
  21. 2026C3ASD: Multi-Level Consistency-Driven Representation Learning for Robust Active Speaker Detectioncited by 0
  22. 2026Talking to Me or Someone Else? Rethinking Talk-to-Me Detection in Egocentric Videoscited by 0
  23. 2026ROAM-ASD: Robust Open-World Active Speaker Detection with Flexible Multimodal Fusioncited by 0
  24. 2026CAST-ASD: Causal Attention over Speakers and Time for Online Active Speaker Detectioncited by 0