mlsanityData to 7 Oct 2026

Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection

arXiv 2107.06592 ↗First posted 14 Jul 2021eess.AS cs.SD eess.IVcited by 110

Not on the map. Why?

Abstract (arXiv)

Active speaker detection (ASD) seeks to detect who is speaking in a visual scene of one or more speakers. The successful ASD depends on accurate interpretation of short-term and long-term audio and visual information, as well as audio-visual interaction. Unlike the prior work where systems make decision instantaneously using short-term features, we propose a novel framework, named TalkNet, that makes decision by taking both short-term and long-term features into consideration. TalkNet consists of audio and visual temporal encoders for feature representation, audio-visual cross-attention mechanism for inter-modality interaction, and a self-attention mechanism to capture long-term speaking evidence. The experiments demonstrate that TalkNet achieves 3.5% and 2.2% improvement over the state-of-the-art systems on the AVA-ActiveSpeaker dataset and Columbia ASD dataset, respectively. Code has been made available at: https://github.com/TaoRuijie/TalkNet_ASD.

Key facts. Cited by 110 papers in the map (data through 7 Oct 2026). First posted on arXiv 14 Jul 2021.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 20; 2023: 24; 2024: 16; 2025: 24; 2026: 22.

110 papers.

Sort:

By area:

  1. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  2. 2023Seeing What You Said: Talking Face Generation Guided by a Lip Reading Expertcited by 98
  3. 2023Self-Supervised Video Forensics by Audio-Visual Anomaly Detectioncited by 84
  4. 2021Sub-word Level Lip Reading With Visual Attentioncited by 60
  5. 2022M3ED: Multi-modal Multi-scene Multi-label Emotional Dialogue Databasecited by 50
  6. 2023RenderMe-360: A Large Digital Asset Library and Benchmarks Towards High-fidelity Head Avatarscited by 45
  7. 2022Make Acoustic and Visual Cues Matter: CH-SIMS v2.0 Dataset and AV-Mixup Consistent Modulecited by 43
  8. 2025AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Modelscited by 43
  9. 2022MIntRec: A New Dataset for Multimodal Intent Recognitioncited by 41
  10. 2023A Light Weight Model for Active Speaker Detectioncited by 39
  11. 2023Affective Faces for Goal-Driven Dyadic Communicationcited by 34
  12. 2023Novel-View Acoustic Synthesiscited by 33
38 more
  1. 2022Learning Long-Term Spatial-Temporal Graphs for Active Speaker Detectioncited by 29
  2. 2023SMILE: Multimodal Dataset for Understanding Laughter in Video with Language Modelscited by 28
  3. 2024MultiTalk: Enhancing 3D Talking Head Generation Across Languages with Multilingual Video Datasetcited by 28
  4. 2022Egocentric Deep Multi-Channel Audio-Visual Active Speaker Localizationcited by 27
  5. 2023LaughTalk: Expressive 3D Talking Head Generation with Laughtercited by 27
  6. 2024MIntRec2.0: A Large-scale Benchmark Dataset for Multimodal Intent Recognition and Out-of-scope Detection in Conversationscited by 25
  7. 2021AVA-AVD: Audio-Visual Speaker Diarization in the Wildcited by 24
  8. 2022End-to-End Active Speaker Detectioncited by 24
  9. 2023LoCoNet: Long-Short Context Network for Active Speaker Detectioncited by 24
  10. 2025StreamAvatar: Streaming Diffusion Models for Real-Time Interactive Human Avatarscited by 24
  11. 2024FlowAVSE: Efficient Audio-Visual Speech Enhancement with Conditional Flow Matchingcited by 23
  12. 2022Look&Listen: Multi-Modal Correlation Learning for Active Speaker Detection and Speech Enhancementcited by 20
  13. 2023Listen to Look into the Future: Audio-Visual Egocentric Gaze Anticipationcited by 20
  14. 2022Egocentric Video Task Translationcited by 19
  15. 2023Sound Localization from Motion: Jointly Learning Sound Direction and Camera Rotationcited by 18
  16. 2023Egocentric Auditory Attention Localization in Conversationscited by 18
  17. 2023AdVerb: Visually Guided Audio Dereverberationcited by 15
  18. 2023TalkNCE: Improving Active Speaker Detection with Talk-Aware Contrastive Learningcited by 15
  19. 2023AV-MaskEnhancer: Enhancing Video Representations through Audio-Visual Masked Autoencodercited by 11
  20. 2026LPM 1.0: Video-based Character Performance Modelcited by 11
  21. 2025VoiceCraft-Dub: Automated Video Dubbing with Neural Codec Language Modelscited by 10
  22. 2025Emotion-Qwen: A Unified Framework for Emotion and Vision Understandingcited by 10
  23. 2025Predicting Turn-Taking and Backchannel in Human-Machine Conversations Using Linguistic, Acoustic, and Visual Signalscited by 10
  24. 2022Rethinking Audio-visual Synchronization for Active Speaker Detectioncited by 9
  25. 2022Intel Labs at Ego4D Challenge 2022: A Better Baseline for Audio-Visual Diarizationcited by 9
  26. 2023Seeing Through the Conversation: Audio-Visual Speech Separation based on Diffusion Modelcited by 9
  27. 2024Friends-MMC: A Dataset for Multi-modal Multi-party Conversation Understandingcited by 9
  28. 2025MSE-Adapter: A Lightweight Plugin Endowing LLMs with the Capability to Perform Multimodal Sentiment Analysis and Emotion Recognitioncited by 9
  29. 2025MAVOS-DD: Multilingual Audio-Video Open-Set Deepfake Detection Benchmarkcited by 9
  30. 2022Distance Matters in Human-Object Interaction Detectioncited by 8
  31. 2023Learning Spatial Features from Audio-Visual Correspondence in Egocentric Videoscited by 8
  32. 2023STHG: Spatial-Temporal Heterogeneous Graph Learning for Advanced Audio-Visual Diarizationcited by 6
  33. 2025LASER: Lip Landmark Assisted Speaker Detection for Robustnesscited by 6
  34. 2025ViBES: A Conversational Agent with Behaviorally-Intelligent 3D Virtual Bodycited by 6
  35. 2022Speaker Extraction with Co-Speech Gestures Cuecited by 5
  36. 2023WASD: A Wilder Active Speaker Detection Datasetcited by 5
  37. 2024Spherical World-Locking for Audio-Visual Localization in Egocentric Videoscited by 5
  38. 2024ASDnB: Merging Face with Body Cues For Robust Active Speaker Detectioncited by 5

50 most cited of 110 All 110 (JSON)