# Wavesplit: End-to-End Speech Separation by Speaker Clustering

arXiv: https://arxiv.org/abs/2002.08933 · HTML: https://mlsanity.com/paper/2002.08933/

## Key facts

- Cited by 75 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Feb 2020.

## Built on by (most cited first, 30 of 75)

- [Dual-path Mamba: Short and Long-term Bidirectional Selective Structured State Space Models for Speech Separation](https://mlsanity.com/paper/2403.18257/) (2024)
- [Diffusion-based Generative Speech Source Separation](https://mlsanity.com/paper/2210.17327/) (2022)
- [MambaMixer: Efficient Selective State Space Models with Dual Token and Channel Selection](https://mlsanity.com/paper/2403.19888/) (2024)
- [SepPrune: Structured Pruning for Efficient Deep Speech Separation](https://mlsanity.com/paper/2505.12079/) (2025)
- [MossFormer: Pushing the Performance Limit of Monaural Speech Separation using Gated Single-Head Transformer with Convolution-Augmented Joint Self-Attentions](https://mlsanity.com/paper/2302.11824/) (2023)
- [RemixIT: Continual self-training of speech enhancement models via bootstrapped remixing](https://mlsanity.com/paper/2202.08862/) (2022)
- [Orion-MSP: Multi-Scale Sparse Attention for Tabular In-Context Learning](https://mlsanity.com/paper/2511.02818/) (2025)
- [SepIt: Approaching a Single Channel Speech Separation Bound](https://mlsanity.com/paper/2205.11801/) (2022)
- [Compute and memory efficient universal sound source separation](https://mlsanity.com/paper/2103.02644/) (2021)
- [Towards Neural Diarization for Unlimited Numbers of Speakers Using Global and Local Attractors](https://mlsanity.com/paper/2107.01545/) (2021)
- [Separate And Diffuse: Using a Pretrained Diffusion Model for Improving Source Separation](https://mlsanity.com/paper/2301.10752/) (2023)
- [Many-Speakers Single Channel Speech Separation with Optimal Permutation Training](https://mlsanity.com/paper/2104.08955/) (2021)
- [IIANet: An Intra- and Inter-Modality Attention Network for Audio-Visual Speech Separation](https://mlsanity.com/paper/2308.08143/) (2023)
- [Sandglasset: A Light Multi-Granularity Self-attentive Network For Time-Domain Speech Separation](https://mlsanity.com/paper/2103.00819/) (2021)
- [Estimating the electrical power output of industrial devices with end-to-end time-series classification in the presence of label noise](https://mlsanity.com/paper/2105.00349/) (2021)
- [Unifying Speech Enhancement and Separation with Gradient Modulation for End-to-End Noise-Robust Speech Separation](https://mlsanity.com/paper/2302.11131/) (2023)
- [Teacher-Student MixIT for Unsupervised and Semi-supervised Speech Separation](https://mlsanity.com/paper/2106.07843/) (2021)
- [Noise-robust Speech Separation with Fast Generative Correction](https://mlsanity.com/paper/2406.07461/) (2024)
- [Weight, Block or Unit? Exploring Sparsity Tradeoffs for Speech Enhancement on Tiny Neural Accelerators](https://mlsanity.com/paper/2111.02351/) (2021)
- [Online Neural Diarization of Unlimited Numbers of Speakers Using Global and Local Attractors](https://mlsanity.com/paper/2206.02432/) (2022)
- [OmniSep: Unified Omni-Modality Sound Separation with Query-Mixup](https://mlsanity.com/paper/2410.21269/) (2024)
- [REAL-M: Towards Speech Separation on Real Mixtures](https://mlsanity.com/paper/2110.10812/) (2021)
- [MixCycle: Unsupervised Speech Separation via Cyclic Mixture Permutation Invariant Training](https://mlsanity.com/paper/2202.03875/) (2022)
- [X-SepFormer: End-to-end Speaker Extraction Network with Explicit Optimization on Speaker Confusion](https://mlsanity.com/paper/2303.05023/) (2023)
- [A Neural State-Space Model Approach to Efficient Speech Separation](https://mlsanity.com/paper/2305.16932/) (2023)
- [DIVE: End-to-end Speech Diarization via Iterative Speaker Embedding](https://mlsanity.com/paper/2105.13802/) (2021)
- [Directed Speech Separation for Automatic Speech Recognition of Long Form Conversational Speech](https://mlsanity.com/paper/2112.05863/) (2021)
- [Speaker Extraction with Co-Speech Gestures Cue](https://mlsanity.com/paper/2203.16840/) (2022)
- [SepMamba: State-space models for speaker separation using Mamba](https://mlsanity.com/paper/2410.20997/) (2024)
- [On permutation invariant training for speech source separation](https://mlsanity.com/paper/2102.04945/) (2021)

## Abstract (arXiv)

We introduce Wavesplit, an end-to-end source separation system. From a single mixture, the model infers a representation for each source and then estimates each source signal given the inferred representations. The model is trained to jointly perform both tasks from the raw waveform. Wavesplit infers a set of source representations via clustering, which addresses the fundamental permutation problem of separation. For speech separation, our sequence-wide speaker representations provide a more robust separation of long, challenging recordings compared to prior work. Wavesplit redefines the state-of-the-art on clean mixtures of 2 or 3 speakers (WSJ0-2/3mix), as well as in noisy and reverberated settings (WHAM/WHAMR). We also set a new benchmark on the recent LibriMix dataset. Finally, we show that Wavesplit is also applicable to other domains, by separating fetal and maternal heart rates from a single abdominal electrocardiogram.