# Towards Low-Latency Tracking of Multiple Speakers With Short-Context Speaker Embeddings

arXiv: https://arxiv.org/abs/2508.14115 · HTML: https://mlsanity.com/paper/2508.14115/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 25 references that are in the map.
- First posted on arXiv 18 Aug 2025.

## Builds on (most cited first, 18 of 18)

- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020)
- [HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking](https://mlsanity.com/paper/2009.07736/) (2020)
- [SpeechBrain: A General-Purpose Speech Toolkit](https://mlsanity.com/paper/2106.04624/) (2021)
- [Continuous speech separation: dataset and analysis](https://mlsanity.com/paper/2001.11482/) (2020)
- [A Survey of Sound Source Localization with Deep Learning Methods](https://mlsanity.com/paper/2109.03465/) (2021)
- [Neural Target Speech Extraction: An Overview](https://mlsanity.com/paper/2301.13341/) (2023)
- [The LOCATA Challenge: Acoustic Source Localization and Tracking](https://mlsanity.com/paper/1909.01008/) (2019)
- [Deep Learning based Multi-Source Localization with Source Splitting and its Effectiveness in Multi-Talker Speech Recognition](https://mlsanity.com/paper/2102.07955/) (2021)
- [Self-supervised speaker embeddings](https://mlsanity.com/paper/1904.03486/) (2019)
- [Open-set Short Utterance Forensic Speaker Verification using Teacher-Student Network with Explicit Inductive Bias](https://mlsanity.com/paper/2009.09556/) (2020)
- [In search of strong embedding extractors for speaker diarisation](https://mlsanity.com/paper/2210.14682/) (2022)
- [Location-based training for multi-channel talker-independent speaker separation](https://mlsanity.com/paper/2110.04289/) (2021)
- [Online Localization and Tracking of Multiple Moving Speakers in Reverberant Environments](https://mlsanity.com/paper/1809.10936/) (2018)
- [Position tracking of a varying number of sound sources with sliding permutation invariant training](https://mlsanity.com/paper/2210.14536/) (2022)
- [Tracking of Intermittent and Moving Speakers : Dataset and Metrics](https://mlsanity.com/paper/2506.11145/) (2025)
- [Speaker Embeddings to Improve Tracking of Intermittent and Moving Speakers](https://mlsanity.com/paper/2506.19875/) (2025)
- [A Teacher-Student approach for extracting informative speaker embeddings from speech mixtures](https://mlsanity.com/paper/2306.00634/) (2023)

## Abstract (arXiv)

Speaker embeddings are promising identity-related features that can enhance the identity assignment performance of a tracking system by leveraging its spatial predictions, i.e, by performing identity reassignment. Common speaker embedding extractors usually struggle with short temporal contexts and overlapping speech, which imposes long-term identity reassignment to exploit longer temporal contexts. However, this increases the probability of tracking system errors, which in turn impacts negatively on identity reassignment. To address this, we propose a Knowledge Distillation (KD) based training approach for short context speaker embedding extraction from two speaker mixtures. We leverage the spatial information of the speaker of interest using beamforming to reduce overlap. We study the feasibility of performing identity reassignment over blocks of fixed size, i.e., blockwise identity reassignment, to go towards a low-latency speaker embedding based tracking system. Results demonstrate that our distilled models are effective at short-context embedding extraction and more robust to overlap. Although, blockwise reassignment results indicate that further work is needed to handle simultaneous speech more effectively.