# Learning Long-Term Spatial-Temporal Graphs for Active Speaker Detection

arXiv: https://arxiv.org/abs/2207.07783 · HTML: https://mlsanity.com/paper/2207.07783/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 41 references that are in the map.
- First posted on arXiv 15 Jul 2022.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Fast Graph Representation Learning with PyTorch Geometric](https://mlsanity.com/paper/1903.02428/) (2019)
- [Dynamic Graph CNN for Learning on Point Clouds](https://mlsanity.com/paper/1801.07829/) (2018)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [Speaker Recognition from Raw Waveform with SincNet](https://mlsanity.com/paper/1808.00158/) (2018)
- [Return of the Devil in the Details: Delving Deep into Convolutional Nets](https://mlsanity.com/paper/1405.3531/) (2014)
- [Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection](https://mlsanity.com/paper/2107.06592/) (2021)
- [EGO-TOPO: Environment Affordances from Egocentric Video](https://mlsanity.com/paper/2001.04583/) (2020)
- [The Conversation: Deep Audio-Visual Speech Enhancement](https://mlsanity.com/paper/1804.04121/) (2018)
- [AVA-ActiveSpeaker: An Audio-Visual Dataset for Active Speaker Detection](https://mlsanity.com/paper/1901.01342/) (2019)
- [Combining Residual Networks with LSTMs for Lipreading](https://mlsanity.com/paper/1703.04105/) (2017)
- [Resource Efficient 3D Convolutional Neural Networks](https://mlsanity.com/paper/1904.02422/) (2019)
- [A Structured Model For Action Detection](https://mlsanity.com/paper/1812.03544/) (2018)
- [Active Speakers in Context](https://mlsanity.com/paper/2005.09812/) (2020)
- [LoGAN: Latent Graph Co-Attention Network for Weakly-Supervised Video Moment Retrieval](https://mlsanity.com/paper/1909.13784/) (2019)
- [Unified Graph Structured Models for Video Understanding](https://mlsanity.com/paper/2103.15662/) (2021)
- [MAAS: Multi-modal Assignation for Active Speaker Detection](https://mlsanity.com/paper/2101.03682/) (2021)
- [Space-Time Crop & Attend: Improving Cross-modal Video Representation Learning](https://mlsanity.com/paper/2103.10211/) (2021)
- [How to Design a Three-Stage Architecture for Audio-Visual Active Speaker Detection in the Wild](https://mlsanity.com/paper/2106.03932/) (2021)
- [UniCon: Unified Context Network for Robust Active Speaker Detection](https://mlsanity.com/paper/2108.02607/) (2021)
- [Naver at ActivityNet Challenge 2019 -- Task B Active Speaker Detection (AVA)](https://mlsanity.com/paper/1906.10555/) (2019)
- [Cross-modal Supervision for Learning Active Speaker Detection in Video](https://mlsanity.com/paper/1603.08907/) (2016)
- [Dynamic Graph Representation Learning for Video Dialog via Multi-Modal Shuffled Transformers](https://mlsanity.com/paper/2007.03848/) (2020)
- [Dynamic Emotion Modeling with Learnable Graphs and Graph Inception Network](https://mlsanity.com/paper/2008.02661/) (2020)

## Built on by (most cited first, 30 of 30)

- [A Light Weight Model for Active Speaker Detection](https://mlsanity.com/paper/2303.04439/) (2023)
- [LoCoNet: Long-Short Context Network for Active Speaker Detection](https://mlsanity.com/paper/2301.08237/) (2023)
- [Listen to Look into the Future: Audio-Visual Egocentric Gaze Anticipation](https://mlsanity.com/paper/2305.03907/) (2023)
- [Egocentric Auditory Attention Localization in Conversations](https://mlsanity.com/paper/2303.16024/) (2023)
- [TalkNCE: Improving Active Speaker Detection with Talk-Aware Contrastive Learning](https://mlsanity.com/paper/2309.12306/) (2023)
- [Joint Audio-Visual Idling Vehicle Detection with Streamlined Input Dependencies](https://mlsanity.com/paper/2410.21170/) (2024)
- [Intel Labs at Ego4D Challenge 2022: A Better Baseline for Audio-Visual Diarization](https://mlsanity.com/paper/2210.07764/) (2022)
- [Learning Spatial Features from Audio-Visual Correspondence in Egocentric Videos](https://mlsanity.com/paper/2307.04760/) (2023)
- [WASD: A Wilder Active Speaker Detection Dataset](https://mlsanity.com/paper/2303.05321/) (2023)
- [STHG: Spatial-Temporal Heterogeneous Graph Learning for Advanced Audio-Visual Diarization](https://mlsanity.com/paper/2306.10608/) (2023)
- [Revisiting Active Speaker Detection: An In-the-Wild Benchmark for Generalization and Robustness](https://mlsanity.com/paper/2505.21954/) (2025)
- [BIAS: A Body-based Interpretable Active Speaker Approach](https://mlsanity.com/paper/2412.05150/) (2024)
- [ASDnB: Merging Face with Body Cues For Robust Active Speaker Detection](https://mlsanity.com/paper/2412.08594/) (2024)
- [Audio-Visual Activity Guided Cross-Modal Identity Association for Active Speaker Detection](https://mlsanity.com/paper/2212.00539/) (2022)
- [VideoSAGE: Video Summarization with Graph Representation Learning](https://mlsanity.com/paper/2404.10539/) (2024)
- [Push-Pull: Characterizing the Adversarial Robustness for Audio-Visual Active Speaker Detection](https://mlsanity.com/paper/2210.00753/) (2022)
- [Whose Emotion Matters? Speaking Activity Localisation without Prior Knowledge](https://mlsanity.com/paper/2211.15377/) (2022)
- [A Novel Framework for Multi-Person Temporal Gaze Following and Social Gaze Prediction](https://mlsanity.com/paper/2403.10511/) (2024)
- [Robust Active Speaker Detection in Noisy Environments](https://mlsanity.com/paper/2403.19002/) (2024)
- [AnnoTheia: A Semi-Automatic Annotation Toolkit for Audio-Visual Speech Technologies](https://mlsanity.com/paper/2402.13152/) (2024)
- [Efficient Audiovisual Speech Processing via MUTUD: Multimodal Training and Unimodal Deployment](https://mlsanity.com/paper/2501.18157/) (2025)
- [TEn-CATG:Text-Enriched Audio-Visual Video Parsing with Multi-Scale Category-Aware Temporal Graph](https://mlsanity.com/paper/2509.04086/) (2025)
- [GateFusion: Hierarchical Gated Cross-Modal Fusion for Active Speaker Detection](https://mlsanity.com/paper/2512.15707/) (2025)
- [GEM-VPC: A dual Graph-Enhanced Multimodal integration for Video Paragraph Captioning](https://mlsanity.com/paper/2410.09377/) (2024)
- [FabuLight-ASD: Unveiling Speech Activity via Body Language](https://mlsanity.com/paper/2411.13674/) (2024)
- [Graph-Based Multimodal and Multi-view Alignment for Keystep Recognition](https://mlsanity.com/paper/2501.04121/) (2025)
- [Keystep Recognition using Graph Neural Networks](https://mlsanity.com/paper/2506.01102/) (2025)
- [EgoVIS@CVPR: PAIR-Net: Enhancing Egocentric Speaker Detection via Pretrained Audio-Visual Fusion and Alignment Loss](https://mlsanity.com/paper/2506.02247/) (2025)
- [Talking to Me or Someone Else? Rethinking Talk-to-Me Detection in Egocentric Videos](https://mlsanity.com/paper/2609.14118/) (2026)
- [CAST-ASD: Causal Attention over Speakers and Time for Online Active Speaker Detection](https://mlsanity.com/paper/2610.14676/) (2026)

## Abstract (arXiv)

Active speaker detection (ASD) in videos with multiple speakers is a challenging task as it requires learning effective audiovisual features and spatial-temporal correlations over long temporal windows. In this paper, we present SPELL, a novel spatial-temporal graph learning framework that can solve complex tasks such as ASD. To this end, each person in a video frame is first encoded in a unique node for that frame. Nodes corresponding to a single person across frames are connected to encode their temporal dynamics. Nodes within a frame are also connected to encode inter-person relationships. Thus, SPELL reduces ASD to a node classification task. Importantly, SPELL is able to reason over long temporal contexts for all nodes without relying on computationally expensive fully connected graph neural networks. Through extensive experiments on the AVA-ActiveSpeaker dataset, we demonstrate that learning graph-based representations can significantly improve the active speaker detection performance owing to its explicit spatial and temporal structure. SPELL outperforms all previous state-of-the-art approaches while requiring significantly lower memory and computational resources. Our code is publicly available at https://github.com/SRA2/SPELL