# Speaker Embeddings to Improve Tracking of Intermittent and Moving Speakers

arXiv: https://arxiv.org/abs/2506.19875 · HTML: https://mlsanity.com/paper/2506.19875/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 34 references that are in the map.
- First posted on arXiv 23 Jun 2025.

## Builds on (most cited first, 19 of 19)

- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020)
- [HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking](https://mlsanity.com/paper/2009.07736/) (2020)
- [SpeechBrain: A General-Purpose Speech Toolkit](https://mlsanity.com/paper/2106.04624/) (2021)
- [Pyroomacoustics: A Python package for audio room simulations and array processing algorithms](https://mlsanity.com/paper/1710.04196/) (2017)
- [A Review of Speaker Diarization: Recent Advances with Deep Learning](https://mlsanity.com/paper/2101.09624/) (2021)
- [Continuous speech separation: dataset and analysis](https://mlsanity.com/paper/2001.11482/) (2020)
- [Integrating end-to-end neural and clustering-based diarization: Getting the best of both worlds](https://mlsanity.com/paper/2010.13366/) (2020)
- [The LOCATA Challenge: Acoustic Source Localization and Tracking](https://mlsanity.com/paper/1909.01008/) (2019)
- [Deep Learning based Multi-Source Localization with Source Splitting and its Effectiveness in Multi-Talker Speech Recognition](https://mlsanity.com/paper/2102.07955/) (2021)
- [Differentiable Tracking-Based Training of Deep Learning Sound Source Localizers](https://mlsanity.com/paper/2111.00030/) (2021)
- [Improved feature extraction for CRNN-based multiple sound source localization](https://mlsanity.com/paper/2105.01897/) (2021)
- [Online Localization and Tracking of Multiple Moving Speakers in Reverberant Environments](https://mlsanity.com/paper/1809.10936/) (2018)
- [TRAMP: Tracking by a Real-time AMbisonic-based Particle filter](https://mlsanity.com/paper/1810.04080/) (2018)
- [Position tracking of a varying number of sound sources with sliding permutation invariant training](https://mlsanity.com/paper/2210.14536/) (2022)
- [Improving Speaker Assignment in Speaker-Attributed ASR for Real Meeting Applications](https://mlsanity.com/paper/2403.06570/) (2024)
- [Dilated U-net based approach for multichannel speech enhancement from First-Order Ambisonics recordings](https://mlsanity.com/paper/2006.01708/) (2020)
- [How to Leverage DNN-based speech enhancement for multi-channel speaker verification?](https://mlsanity.com/paper/2210.08834/) (2022)
- [Tracking of Intermittent and Moving Speakers : Dataset and Metrics](https://mlsanity.com/paper/2506.11145/) (2025)

## Built on by (most cited first, 2 of 2)

- [Towards Low-Latency Tracking of Multiple Speakers With Short-Context Speaker Embeddings](https://mlsanity.com/paper/2508.14115/) (2025)
- [Weakly Guided and Autoregressive Beamformer Parameterization for Generalizable Moving Speaker Extraction in Higher-Order Ambisonics](https://mlsanity.com/paper/2607.04471/) (2026)

## Abstract (arXiv)

Speaker tracking methods often rely on spatial observations to assign coherent track identities over time. This raises limits in scenarios with intermittent and moving speakers, i.e., speakers that may change position when they are inactive, thus leading to discontinuous spatial trajectories. This paper proposes to investigate the use of speaker embeddings, in a simple solution to this issue. We propose to perform identity reassignment post-tracking, using speaker embeddings. We leverage trajectory-related information provided by an initial tracking step and multichannel audio signal. Beamforming is used to enhance the signal towards the speakers' positions in order to compute speaker embeddings. These are then used to assign new track identities based on an enrollment pool. We evaluate the performance of the proposed speaker embedding-based identity reassignment method on a dataset where speakers change position during inactivity periods. Results show that it consistently improves the identity assignment performance of neural and standard tracking systems. In particular, we study the impact of beamforming and input duration for embedding extraction.