# Single-Channel Multi-Speaker Separation using Deep Clustering

arXiv: https://arxiv.org/abs/1607.02173 · HTML: https://mlsanity.com/paper/1607.02173/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jul 2016.

## Built on by (most cited first, 30 of 45)

- [Streaming Multi-Talker ASR with Token-Level Serialized Output Training](https://mlsanity.com/paper/2202.00842/) (2022)
- [Visual Scene Graphs for Audio Source Separation](https://mlsanity.com/paper/2109.11955/) (2021)
- [Real-Time Target Sound Extraction](https://mlsanity.com/paper/2211.02250/) (2022)
- [Personalized Speech Enhancement: New Models and Comprehensive Evaluation](https://mlsanity.com/paper/2110.09625/) (2021)
- [Instruct-MusicGen: Unlocking Text-to-Music Editing for Music Language Models via Instruction Tuning](https://mlsanity.com/paper/2405.18386/) (2024)
- [Heterogeneous Target Speech Separation](https://mlsanity.com/paper/2204.03594/) (2022)
- [Many-Speakers Single Channel Speech Separation with Optimal Permutation Training](https://mlsanity.com/paper/2104.08955/) (2021)
- [Discretization and Re-synthesis: an alternative method to solve the Cocktail Party Problem](https://mlsanity.com/paper/2112.09382/) (2021)
- [Multi-turn RNN-T for streaming recognition of multi-party speech](https://mlsanity.com/paper/2112.10200/) (2021)
- [Teacher-Student MixIT for Unsupervised and Semi-supervised Speech Separation](https://mlsanity.com/paper/2106.07843/) (2021)
- [SoloSpeech: Enhancing Intelligibility and Quality in Target Speech Extraction through a Cascaded Generative Pipeline](https://mlsanity.com/paper/2505.19314/) (2025)
- [Weight, Block or Unit? Exploring Sparsity Tradeoffs for Speech Enhancement on Tiny Neural Accelerators](https://mlsanity.com/paper/2111.02351/) (2021)
- [Target Speaker Voice Activity Detection with Transformers and Its Integration with End-to-End Neural Diarization](https://mlsanity.com/paper/2208.13085/) (2022)
- [Latent Iterative Refinement for Modular Source Separation](https://mlsanity.com/paper/2211.11917/) (2022)
- [TriBERT: Full-body Human-centric Audio-visual Representation Learning for Visual Sound Separation](https://mlsanity.com/paper/2110.13412/) (2021)
- [X-SepFormer: End-to-end Speaker Extraction Network with Explicit Optimization on Speaker Confusion](https://mlsanity.com/paper/2303.05023/) (2023)
- [Typing to Listen at the Cocktail Party: Text-Guided Target Speaker Extraction](https://mlsanity.com/paper/2310.07284/) (2023)
- [Optimal Condition Training for Target Source Separation](https://mlsanity.com/paper/2211.05927/) (2022)
- [AudioSlots: A slot-centric generative model for audio separation](https://mlsanity.com/paper/2305.05591/) (2023)
- [SepMamba: State-space models for speaker separation using Mamba](https://mlsanity.com/paper/2410.20997/) (2024)
- [Stepwise-Refining Speech Separation Network via Fine-Grained Encoding in High-order Latent Domain](https://mlsanity.com/paper/2110.04791/) (2021)
- [Mixture Encoder for Joint Speech Separation and Recognition](https://mlsanity.com/paper/2306.12173/) (2023)
- [A Study of the Scale Invariant Signal to Distortion Ratio in Speech Separation with Noisy References](https://mlsanity.com/paper/2508.14623/) (2025)
- [Separator-Transducer-Segmenter: Streaming Recognition and Segmentation of Multi-party Speech](https://mlsanity.com/paper/2205.05199/) (2022)
- [Deformable Temporal Convolutional Networks for Monaural Noisy Reverberant Speech Separation](https://mlsanity.com/paper/2210.15305/) (2022)
- [Deep neural network techniques for monaural speech enhancement: state of the art analysis](https://mlsanity.com/paper/2212.00369/) (2022)
- [On Time Domain Conformer Models for Monaural Speech Separation in Noisy Reverberant Acoustic Environments](https://mlsanity.com/paper/2310.06125/) (2023)
- [SCENEBench: An Audio Understanding Benchmark Grounded in Assistive and Industrial Use Cases](https://mlsanity.com/paper/2603.09853/) (2026)
- [Audio-Visual Speech Separation Using Cross-Modal Correspondence Loss](https://mlsanity.com/paper/2103.01463/) (2021)
- [Reverberation as Supervision for Speech Separation](https://mlsanity.com/paper/2211.08303/) (2022)

## Abstract (arXiv)

Deep clustering is a recently introduced deep learning architecture that uses discriminatively trained embeddings as the basis for clustering. It was recently applied to spectrogram segmentation, resulting in impressive results on speaker-independent multi-speaker separation. In this paper we extend the baseline system with an end-to-end signal approximation objective that greatly improves performance on a challenging speech separation. We first significantly improve upon the baseline system performance by incorporating better regularization, larger temporal context, and a deeper architecture, culminating in an overall improvement in signal to distortion ratio (SDR) of 10.3 dB compared to the baseline of 6.0 dB for two-speaker separation, as well as a 7.1 dB SDR improvement for three-speaker separation. We then extend the model to incorporate an enhancement layer to refine the signal estimates, and perform end-to-end training through both the clustering and enhancement stages to maximize signal fidelity. We evaluate the results using automatic speech recognition. The new signal approximation objective, combined with end-to-end training, produces unprecedented performance, reducing the word error rate (WER) from 89.1% down to 30.8%. This represents a major advancement towards solving the cocktail party problem.