# Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection

arXiv: https://arxiv.org/abs/2107.06592 · HTML: https://mlsanity.com/paper/2107.06592/

## Key facts

- Cited by 113 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jul 2021.

## Built on by (most cited first, 30 of 113)

- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [Seeing What You Said: Talking Face Generation Guided by a Lip Reading Expert](https://mlsanity.com/paper/2303.17480/) (2023)
- [Self-Supervised Video Forensics by Audio-Visual Anomaly Detection](https://mlsanity.com/paper/2301.01767/) (2023)
- [Sub-word Level Lip Reading With Visual Attention](https://mlsanity.com/paper/2110.07603/) (2021)
- [M3ED: Multi-modal Multi-scene Multi-label Emotional Dialogue Database](https://mlsanity.com/paper/2205.10237/) (2022)
- [RenderMe-360: A Large Digital Asset Library and Benchmarks Towards High-fidelity Head Avatars](https://mlsanity.com/paper/2305.13353/) (2023)
- [Make Acoustic and Visual Cues Matter: CH-SIMS v2.0 Dataset and AV-Mixup Consistent Module](https://mlsanity.com/paper/2209.02604/) (2022)
- [AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models](https://mlsanity.com/paper/2501.16566/) (2025)
- [MIntRec: A New Dataset for Multimodal Intent Recognition](https://mlsanity.com/paper/2209.04355/) (2022)
- [A Light Weight Model for Active Speaker Detection](https://mlsanity.com/paper/2303.04439/) (2023)
- [Affective Faces for Goal-Driven Dyadic Communication](https://mlsanity.com/paper/2301.10939/) (2023)
- [Novel-View Acoustic Synthesis](https://mlsanity.com/paper/2301.08730/) (2023)
- [Learning Long-Term Spatial-Temporal Graphs for Active Speaker Detection](https://mlsanity.com/paper/2207.07783/) (2022)
- [SMILE: Multimodal Dataset for Understanding Laughter in Video with Language Models](https://mlsanity.com/paper/2312.09818/) (2023)
- [MultiTalk: Enhancing 3D Talking Head Generation Across Languages with Multilingual Video Dataset](https://mlsanity.com/paper/2406.14272/) (2024)
- [Egocentric Deep Multi-Channel Audio-Visual Active Speaker Localization](https://mlsanity.com/paper/2201.01928/) (2022)
- [LaughTalk: Expressive 3D Talking Head Generation with Laughter](https://mlsanity.com/paper/2311.00994/) (2023)
- [MIntRec2.0: A Large-scale Benchmark Dataset for Multimodal Intent Recognition and Out-of-scope Detection in Conversations](https://mlsanity.com/paper/2403.10943/) (2024)
- [AVA-AVD: Audio-Visual Speaker Diarization in the Wild](https://mlsanity.com/paper/2111.14448/) (2021)
- [End-to-End Active Speaker Detection](https://mlsanity.com/paper/2203.14250/) (2022)
- [LoCoNet: Long-Short Context Network for Active Speaker Detection](https://mlsanity.com/paper/2301.08237/) (2023)
- [StreamAvatar: Streaming Diffusion Models for Real-Time Interactive Human Avatars](https://mlsanity.com/paper/2512.22065/) (2025)
- [FlowAVSE: Efficient Audio-Visual Speech Enhancement with Conditional Flow Matching](https://mlsanity.com/paper/2406.09286/) (2024)
- [Look&Listen: Multi-Modal Correlation Learning for Active Speaker Detection and Speech Enhancement](https://mlsanity.com/paper/2203.02216/) (2022)
- [Listen to Look into the Future: Audio-Visual Egocentric Gaze Anticipation](https://mlsanity.com/paper/2305.03907/) (2023)
- [Egocentric Video Task Translation](https://mlsanity.com/paper/2212.06301/) (2022)
- [Sound Localization from Motion: Jointly Learning Sound Direction and Camera Rotation](https://mlsanity.com/paper/2303.11329/) (2023)
- [Egocentric Auditory Attention Localization in Conversations](https://mlsanity.com/paper/2303.16024/) (2023)
- [TalkNCE: Improving Active Speaker Detection with Talk-Aware Contrastive Learning](https://mlsanity.com/paper/2309.12306/) (2023)
- [AdVerb: Visually Guided Audio Dereverberation](https://mlsanity.com/paper/2308.12370/) (2023)

## Abstract (arXiv)

Active speaker detection (ASD) seeks to detect who is speaking in a visual scene of one or more speakers. The successful ASD depends on accurate interpretation of short-term and long-term audio and visual information, as well as audio-visual interaction. Unlike the prior work where systems make decision instantaneously using short-term features, we propose a novel framework, named TalkNet, that makes decision by taking both short-term and long-term features into consideration. TalkNet consists of audio and visual temporal encoders for feature representation, audio-visual cross-attention mechanism for inter-modality interaction, and a self-attention mechanism to capture long-term speaking evidence. The experiments demonstrate that TalkNet achieves 3.5% and 2.2% improvement over the state-of-the-art systems on the AVA-ActiveSpeaker dataset and Columbia ASD dataset, respectively. Code has been made available at: https://github.com/TaoRuijie/TalkNet_ASD.