# BIAS: A Body-based Interpretable Active Speaker Approach

arXiv: https://arxiv.org/abs/2412.05150 · HTML: https://mlsanity.com/paper/2412.05150/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 74 references that are in the map.
- First posted on arXiv 6 Dec 2024.

## Builds on (most cited first, 12 of 12)

- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [SmoothGrad: removing noise by adding noise](https://mlsanity.com/paper/1706.03825/) (2017)
- [Visual Transformers: Token-based Image Representation and Processing for Computer Vision](https://mlsanity.com/paper/2006.03677/) (2020)
- [Not Just a Black Box: Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1605.01713/) (2016)
- [A Richly Annotated Dataset for Pedestrian Attribute Recognition](https://mlsanity.com/paper/1603.07054/) (2016)
- [EasyCom: An Augmented Reality Dataset to Support Algorithms for Easy Communication in Noisy Environments](https://mlsanity.com/paper/2107.04174/) (2021)
- [Learning Long-Term Spatial-Temporal Graphs for Active Speaker Detection](https://mlsanity.com/paper/2207.07783/) (2022)
- [Naver at ActivityNet Challenge 2019 -- Task B Active Speaker Detection (AVA)](https://mlsanity.com/paper/1906.10555/) (2019)
- [End-to-End Active Speaker Detection](https://mlsanity.com/paper/2203.14250/) (2022)
- [LoCoNet: Long-Short Context Network for Active Speaker Detection](https://mlsanity.com/paper/2301.08237/) (2023)
- [Look Who's Talking: Active Speaker Detection in the Wild](https://mlsanity.com/paper/2108.07640/) (2021)
- [Rethinking of Pedestrian Attribute Recognition: Realistic Datasets with Efficient Method](https://mlsanity.com/paper/2005.11909/) (2020)

## Built on by (most cited first, 5 of 5)

- [ASDnB: Merging Face with Body Cues For Robust Active Speaker Detection](https://mlsanity.com/paper/2412.08594/) (2024)
- [LISArD: Learning Image Similarity to Defend Against Gray-box Adversarial Attacks](https://mlsanity.com/paper/2502.20562/) (2025)
- [ZQBA: Zero Query Black-box Adversarial Attack](https://mlsanity.com/paper/2510.00769/) (2025)
- [GateFusion: Hierarchical Gated Cross-Modal Fusion for Active Speaker Detection](https://mlsanity.com/paper/2512.15707/) (2025)
- [Are DeepFakes Realistic Enough? Exploring Semantic Mismatch as a Novel Challenge](https://mlsanity.com/paper/2604.28022/) (2026)

## Abstract (arXiv)

State-of-the-art Active Speaker Detection (ASD) approaches heavily rely on audio and facial features to perform, which is not a sustainable approach in wild scenarios. Although these methods achieve good results in the standard AVA-ActiveSpeaker set, a recent wilder ASD dataset (WASD) showed the limitations of such models and raised the need for new approaches. As such, we propose BIAS, a model that, for the first time, combines audio, face, and body information, to accurately predict active speakers in varying/challenging conditions. Additionally, we design BIAS to provide interpretability by proposing a novel use for Squeeze-and-Excitation blocks, namely in attention heatmaps creation and feature importance assessment. For a full interpretability setup, we annotate an ASD-related actions dataset (ASD-Text) to finetune a ViT-GPT2 for text scene description to complement BIAS interpretability. The results show that BIAS is state-of-the-art in challenging conditions where body-based features are of utmost importance (Columbia, open-settings, and WASD), and yields competitive results in AVA-ActiveSpeaker, where face is more influential than body for ASD. BIAS interpretability also shows the features/aspects more relevant towards ASD prediction in varying settings, making it a strong baseline for further developments in interpretable ASD models, and is available at https://github.com/Tiago-Roxo/BIAS.