# Variational Bayesian Inference for Audio-Visual Tracking of Multiple Speakers

arXiv: https://arxiv.org/abs/1809.10961 · HTML: https://mlsanity.com/paper/1809.10961/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Sep 2018.

## Built on by (most cited first, 10 of 10)

- [TransCenter: Transformers with Dense Representations for Multiple-Object Tracking](https://mlsanity.com/paper/2103.15145/) (2021)
- [Egocentric Auditory Attention Localization in Conversations](https://mlsanity.com/paper/2303.16024/) (2023)
- [Multi-Modal Perception Attention Network with Self-Supervised Learning for Audio-Visual Speaker Tracking](https://mlsanity.com/paper/2112.07423/) (2021)
- [Hierarchical Augmentation and Distillation for Class Incremental Audio-Visual Video Recognition](https://mlsanity.com/paper/2401.06287/) (2024)
- [AcousticFusion: Fusing Sound Source Localization to Visual SLAM in Dynamic Environments](https://mlsanity.com/paper/2108.01246/) (2021)
- [Data Fusion for Audiovisual Speaker Localization: Extending Dynamic Stream Weights to the Spatial Domain](https://mlsanity.com/paper/2102.11588/) (2021)
- [Mixture of Dynamical Variational Autoencoders for Multi-Source Trajectory Modeling and Separation](https://mlsanity.com/paper/2312.04167/) (2023)
- [STNet: Deep Audio-Visual Fusion Network for Robust Speaker Tracking](https://mlsanity.com/paper/2410.05964/) (2024)
- [Unsupervised Multiple-Object Tracking with a Dynamical Variational Autoencoder](https://mlsanity.com/paper/2202.09315/) (2022)
- [A Sparse Bayesian Learning for Diagnosis of Nonstationary and Spatially Correlated Faults with Application to Multistation Assembly Systems](https://mlsanity.com/paper/2310.16058/) (2023)

## Abstract (arXiv)

In this paper we address the problem of tracking multiple speakers via the fusion of visual and auditory information. We propose to exploit the complementary nature of these two modalities in order to accurately estimate smooth trajectories of the tracked persons, to deal with the partial or total absence of one of the modalities over short periods of time, and to estimate the acoustic status -- either speaking or silent -- of each tracked person along time. We propose to cast the problem at hand into a generative audio-visual fusion (or association) model formulated as a latent-variable temporal graphical model. This may well be viewed as the problem of maximizing the posterior joint distribution of a set of continuous and discrete latent variables given the past and current observations, which is intractable. We propose a variational inference model which amounts to approximate the joint distribution with a factorized distribution. The solution takes the form of a closed-form expectation maximization procedure. We describe in detail the inference algorithm, we evaluate its performance and we compare it with several baseline methods. These experiments show that the proposed audio-visual tracker performs well in informal meetings involving a time-varying number of people.