# ASDnB: Merging Face with Body Cues For Robust Active Speaker Detection

arXiv: https://arxiv.org/abs/2412.08594 · HTML: https://mlsanity.com/paper/2412.08594/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 49 references that are in the map.
- First posted on arXiv 11 Dec 2024.

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actions](https://mlsanity.com/paper/1705.08421/) (2017)
- [Self-Supervised Learning of Audio-Visual Objects from Video](https://mlsanity.com/paper/2008.04237/) (2020)
- [S3FD: Single Shot Scale-invariant Face Detector](https://mlsanity.com/paper/1708.05237/) (2017)
- [Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection](https://mlsanity.com/paper/2107.06592/) (2021)
- [Improving Person Re-identification by Attribute and Identity Learning](https://mlsanity.com/paper/1703.07220/) (2017)
- [HydraPlus-Net: Attentive Deep Features for Pedestrian Analysis](https://mlsanity.com/paper/1709.09930/) (2017)
- [AVA-ActiveSpeaker: An Audio-Visual Dataset for Active Speaker Detection](https://mlsanity.com/paper/1901.01342/) (2019)
- [Spot the conversation: speaker diarisation in the wild](https://mlsanity.com/paper/2007.01216/) (2020)
- [Look, Listen, and Attend: Co-Attention Network for Self-Supervised Audio-Visual Representation Learning](https://mlsanity.com/paper/2008.05789/) (2020)
- [Deep Imbalanced Attribute Classification using Visual Attention Aggregation](https://mlsanity.com/paper/1807.03903/) (2018)
- [Improving Pedestrian Attribute Recognition With Weakly-Supervised Multi-Scale Attribute-Specific Localization](https://mlsanity.com/paper/1910.04562/) (2019)
- [Active Speakers in Context](https://mlsanity.com/paper/2005.09812/) (2020)
- [A Light Weight Model for Active Speaker Detection](https://mlsanity.com/paper/2303.04439/) (2023)
- [MAAS: Multi-modal Assignation for Active Speaker Detection](https://mlsanity.com/paper/2101.03682/) (2021)
- [A Richly Annotated Dataset for Pedestrian Attribute Recognition](https://mlsanity.com/paper/1603.07054/) (2016)
- [How to Design a Three-Stage Architecture for Audio-Visual Active Speaker Detection in the Wild](https://mlsanity.com/paper/2106.03932/) (2021)
- [EasyCom: An Augmented Reality Dataset to Support Algorithms for Easy Communication in Noisy Environments](https://mlsanity.com/paper/2107.04174/) (2021)
- [Learning Long-Term Spatial-Temporal Graphs for Active Speaker Detection](https://mlsanity.com/paper/2207.07783/) (2022)
- [UniCon: Unified Context Network for Robust Active Speaker Detection](https://mlsanity.com/paper/2108.02607/) (2021)
- [Naver at ActivityNet Challenge 2019 -- Task B Active Speaker Detection (AVA)](https://mlsanity.com/paper/1906.10555/) (2019)
- [End-to-End Active Speaker Detection](https://mlsanity.com/paper/2203.14250/) (2022)
- [LoCoNet: Long-Short Context Network for Active Speaker Detection](https://mlsanity.com/paper/2301.08237/) (2023)
- [Audio-Visual Speaker Diarization Based on Spatiotemporal Bayesian Fusion](https://mlsanity.com/paper/1603.09725/) (2016)
- [Who said that?: Audio-visual speaker diarisation of real-world meetings](https://mlsanity.com/paper/1906.10042/) (2019)
- [Cross-modal Supervision for Learning Active Speaker Detection in Video](https://mlsanity.com/paper/1603.08907/) (2016)
- [Look Who's Talking: Active Speaker Detection in the Wild](https://mlsanity.com/paper/2108.07640/) (2021)
- [Rethinking of Pedestrian Attribute Recognition: Realistic Datasets with Efficient Method](https://mlsanity.com/paper/2005.11909/) (2020)
- [Multi-target DoA Estimation with an Audio-visual Fusion Mechanism](https://mlsanity.com/paper/2105.06107/) (2021)

## Built on by (most cited first, 5 of 5)

- [LISArD: Learning Image Similarity to Defend Against Gray-box Adversarial Attacks](https://mlsanity.com/paper/2502.20562/) (2025)
- [ZQBA: Zero Query Black-box Adversarial Attack](https://mlsanity.com/paper/2510.00769/) (2025)
- [GateFusion: Hierarchical Gated Cross-Modal Fusion for Active Speaker Detection](https://mlsanity.com/paper/2512.15707/) (2025)
- [EgoVIS@CVPR: PAIR-Net: Enhancing Egocentric Speaker Detection via Pretrained Audio-Visual Fusion and Alignment Loss](https://mlsanity.com/paper/2506.02247/) (2025)
- [Are DeepFakes Realistic Enough? Exploring Semantic Mismatch as a Novel Challenge](https://mlsanity.com/paper/2604.28022/) (2026)

## Abstract (arXiv)

State-of-the-art Active Speaker Detection (ASD) approaches mainly use audio and facial features as input. However, the main hypothesis in this paper is that body dynamics is also highly correlated to "speaking" (and "listening") actions and should be particularly useful in wild conditions (e.g., surveillance settings), where face cannot be reliably accessed. We propose ASDnB, a model that singularly integrates face with body information by merging the inputs at different steps of feature extraction. Our approach splits 3D convolution into 2D and 1D to reduce computation cost without loss of performance, and is trained with adaptive weight feature importance for improved complement of face with body data. Our experiments show that ASDnB achieves state-of-the-art results in the benchmark dataset (AVA-ActiveSpeaker), in the challenging data of WASD, and in cross-domain settings using Columbia. This way, ASDnB can perform in multiple settings, which is positively regarded as a strong baseline for robust ASD models (code available at https://github.com/Tiago-Roxo/ASDnB).