# STNet: Deep Audio-Visual Fusion Network for Robust Speaker Tracking

arXiv: https://arxiv.org/abs/2410.05964 · HTML: https://mlsanity.com/paper/2410.05964/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 68 references that are in the map.
- First posted on arXiv 8 Oct 2024.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [Fully-Convolutional Siamese Networks for Object Tracking](https://mlsanity.com/paper/1606.09549/) (2016)
- [GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wild](https://mlsanity.com/paper/1810.11981/) (2018)
- [Transformer Tracking](https://mlsanity.com/paper/2103.15436/) (2021)
- [Deep Audio-Visual Speech Recognition](https://mlsanity.com/paper/1809.02108/) (2018)
- [Objects that Sound](https://mlsanity.com/paper/1712.06651/) (2017)
- [Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separation](https://mlsanity.com/paper/1804.03619/) (2018)
- [End-to-end Audio-visual Speech Recognition with Conformers](https://mlsanity.com/paper/2102.06657/) (2021)
- [Audio to Body Dynamics](https://mlsanity.com/paper/1712.09382/) (2017)
- [Audio-Visual Speaker Diarization Based on Spatiotemporal Bayesian Fusion](https://mlsanity.com/paper/1603.09725/) (2016)
- [Robust Sound Source Tracking Using SRP-PHAT and 3D Convolutional Neural Networks](https://mlsanity.com/paper/2006.09006/) (2020)
- [Exploiting Audio-Visual Consistency with Partial Supervision for Spatial Audio Generation](https://mlsanity.com/paper/2105.00708/) (2021)
- [Self-Supervised Adaptation of High-Fidelity Face Models for Monocular Performance Tracking](https://mlsanity.com/paper/1907.10815/) (2019)
- [Variational Bayesian Inference for Audio-Visual Tracking of Multiple Speakers](https://mlsanity.com/paper/1809.10961/) (2018)
- [MPN: Multimodal Parallel Network for Audio-Visual Event Localization](https://mlsanity.com/paper/2104.02971/) (2021)
- [Multi-Modal Perception Attention Network with Self-Supervised Learning for Audio-Visual Speaker Tracking](https://mlsanity.com/paper/2112.07423/) (2021)
- [Self-supervised Neural Audio-Visual Sound Source Localization via Probabilistic Spatial Modeling](https://mlsanity.com/paper/2007.13976/) (2020)
- [Detect Faces Efficiently: A Survey and Evaluations](https://mlsanity.com/paper/2112.01787/) (2021)
- [Audio-Visual Speaker Tracking: Progress, Challenges, and Future Directions](https://mlsanity.com/paper/2310.14778/) (2023)

## Built on by (most cited first, 1 of 1)

- [DSGC-Net: A Dual-Stream Graph Convolutional Network for Crowd Counting via Feature Correlation Mining](https://mlsanity.com/paper/2509.02261/) (2025)

## Abstract (arXiv)

Audio-visual speaker tracking aims to determine the location of human targets in a scene using signals captured by a multi-sensor platform, whose accuracy and robustness can be improved by multi-modal fusion methods. Recently, several fusion methods have been proposed to model the correlation in multiple modalities. However, for the speaker tracking problem, the cross-modal interaction between audio and visual signals hasn't been well exploited. To this end, we present a novel Speaker Tracking Network (STNet) with a deep audio-visual fusion model in this work. We design a visual-guided acoustic measurement method to fuse heterogeneous cues in a unified localization space, which employs visual observations via a camera model to construct the enhanced acoustic map. For feature fusion, a cross-modal attention module is adopted to jointly model multi-modal contexts and interactions. The correlated information between audio and visual features is further interacted in the fusion model. Moreover, the STNet-based tracker is applied to multi-speaker cases by a quality-aware module, which evaluates the reliability of multi-modal observations to achieve robust tracking in complex scenarios. Experiments on the AV16.3 and CAV3D datasets show that the proposed STNet-based tracker outperforms uni-modal methods and state-of-the-art audio-visual speaker trackers.