# Robust Target Speaker Diarization and Separation via Augmented Speaker Embedding Sampling

arXiv: https://arxiv.org/abs/2508.06393 · HTML: https://mlsanity.com/paper/2508.06393/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 44 references that are in the map.
- First posted on arXiv 8 Aug 2025.

## Builds on (most cited first, 28 of 28)

- [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020)
- [Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation](https://mlsanity.com/paper/1809.07454/) (2018)
- [Using Pre-Training Can Improve Model Robustness and Uncertainty](https://mlsanity.com/paper/1901.09960/) (2019)
- [NeMo: a toolkit for building AI applications using Neural Modules](https://mlsanity.com/paper/1909.09577/) (2019)
- [Permutation Invariant Training of Deep Models for Speaker-Independent Multi-talker Speech Separation](https://mlsanity.com/paper/1607.00325/) (2016)
- [Exploring the Limits of Large Scale Pre-training](https://mlsanity.com/paper/2110.02095/) (2021)
- [End-to-end speaker segmentation for overlap-aware resegmentation](https://mlsanity.com/paper/2104.04045/) (2021)
- [Continuous speech separation: dataset and analysis](https://mlsanity.com/paper/2001.11482/) (2020)
- [Target-Speaker Voice Activity Detection: a Novel Approach for Multi-Speaker Diarization in a Dinner Party Scenario](https://mlsanity.com/paper/2005.07272/) (2020)
- [Automatic Speech Recognition using Advanced Deep Learning Approaches: A survey](https://mlsanity.com/paper/2403.01255/) (2024)
- [Integration of speech separation, diarization, and recognition for multi-speaker meetings: System description, comparison, and analysis](https://mlsanity.com/paper/2011.02014/) (2020)
- [MeetEval: A Toolkit for Computation of Word Error Rates for Meeting Transcription Systems](https://mlsanity.com/paper/2307.11394/) (2023)
- [Overlap-aware diarization: resegmentation using neural end-to-end overlapped speech detection](https://mlsanity.com/paper/1910.11646/) (2019)
- [ESPnet-se: end-to-end speech enhancement and separation toolkit designed for asr integration](https://mlsanity.com/paper/2011.03706/) (2020)
- [SkiM: Skipping Memory LSTM for Low-Latency Real-Time Continuous Speech Separation](https://mlsanity.com/paper/2201.10800/) (2022)
- [Target-speaker Voice Activity Detection with Improved I-Vector Estimation for Unknown Number of Speaker](https://mlsanity.com/paper/2108.03342/) (2021)
- [Separate and Reconstruct: Asymmetric Encoder-Decoder for Speech Separation](https://mlsanity.com/paper/2406.05983/) (2024)
- [HuBERT-EE: Early Exiting HuBERT for Efficient Speech Recognition](https://mlsanity.com/paper/2204.06328/) (2022)
- [TS-SEP: Joint Diarization and Separation Conditioned on Estimated Speaker Embeddings](https://mlsanity.com/paper/2303.03849/) (2023)
- [The CHiME-8 DASR Challenge for Generalizable and Array Agnostic Distant Automatic Speech Recognition and Diarization](https://mlsanity.com/paper/2407.16447/) (2024)
- [Overview of Speaker Modeling and Its Applications: From the Lens of Deep Speaker Representation Learning](https://mlsanity.com/paper/2407.15188/) (2024)
- [SpeakerBeam-SS: Real-time Target Speaker Extraction with Lightweight Conv-TasNet and State Space Modeling](https://mlsanity.com/paper/2407.01857/) (2024)
- [Compositional embedding models for speaker identification and diarization with simultaneous speech from 2+ speakers](https://mlsanity.com/paper/2010.11803/) (2020)
- [Focus on the Sound around You: Monaural Target Speaker Extraction via Distance and Speaker Information](https://mlsanity.com/paper/2306.16241/) (2023)
- [Efficiently Train ASR Models that Memorize Less and Perform Better with Per-core Clipping](https://mlsanity.com/paper/2406.02004/) (2024)
- [Once more Diarization: Improving meeting transcription systems through segment-level speaker reassignment](https://mlsanity.com/paper/2406.03155/) (2024)
- [Universal Speaker Embedding Free Target Speaker Extraction and Personal Voice Activity Detection](https://mlsanity.com/paper/2501.03612/) (2025)
- [North America Bixby Speaker Diarization System for the VoxCeleb Speaker Recognition Challenge 2021](https://mlsanity.com/paper/2109.13518/) (2021)

## Abstract (arXiv)

Traditional speech separation and speaker diarization approaches rely on prior knowledge of target speakers or a predetermined number of participants in audio signals. To address these limitations, recent advances focus on developing enrollment-free methods capable of identifying targets without explicit speaker labeling. This work introduces a new approach to train simultaneous speech separation and diarization using automatic identification of target speaker embeddings, within mixtures. Our proposed model employs a dual-stage training pipeline designed to learn robust speaker representation features that are resilient to background noise interference. Furthermore, we present an overlapping spectral loss function specifically tailored for enhancing diarization accuracy during overlapped speech frames. Experimental results show significant performance gains compared to the current SOTA baseline, achieving 71% relative improvement in DER and 69% in cpWER.