# Improving Music Source Separation with Diffusion and Consistency Refinement

arXiv: https://arxiv.org/abs/2412.06965 · HTML: https://mlsanity.com/paper/2412.06965/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Audio, speech and music, one of 14 directions in Diffusion models.
- Builds on 48 of its 67 references that are in the map.
- First posted on arXiv 9 Dec 2024.

## Placed in

- Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/)

## Builds on (most cited first, 30 of 48)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [On the Variance of the Adaptive Learning Rate and Beyond](https://mlsanity.com/paper/1908.03265/) (2019)
- [Consistency Trajectory Models: Learning Probability Flow ODE Trajectory of Diffusion](https://mlsanity.com/paper/2310.02279/) (2023)
- [SDR - half-baked or well done?](https://mlsanity.com/paper/1811.02508/) (2018)
- [Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation](https://mlsanity.com/paper/1809.07454/) (2018)
- [Hybrid Transformers for Music Source Separation](https://mlsanity.com/paper/2211.08553/) (2022)
- [Wave-U-Net: A Multi-Scale Neural Network for End-to-End Audio Source Separation](https://mlsanity.com/paper/1806.03185/) (2018)
- [StoRM: A Diffusion-based Stochastic Regeneration Model for Speech Enhancement and Dereverberation](https://mlsanity.com/paper/2212.11851/) (2022)
- [Music Source Separation in the Waveform Domain](https://mlsanity.com/paper/1911.13254/) (2019)
- [Hybrid Spectrogram and Waveform Source Separation](https://mlsanity.com/paper/2111.03600/) (2021)
- [Cutting Music Source Separation Some Slakh: A Dataset to Study the Impact of Training Data Quality and Quantity](https://mlsanity.com/paper/1909.08494/) (2019)
- [Music Source Separation with Band-split RNN](https://mlsanity.com/paper/2209.15174/) (2022)
- [Multi-Source Diffusion Models for Simultaneous Music Generation and Separation](https://mlsanity.com/paper/2302.02257/) (2023)
- [The 2018 Signal Separation Evaluation Campaign](https://mlsanity.com/paper/1804.06267/) (2018)
- [Universal Sound Separation](https://mlsanity.com/paper/1905.03330/) (2019)
- [Diffusion-based Generative Speech Source Separation](https://mlsanity.com/paper/2210.17327/) (2022)
- [Music Source Separation with Band-Split RoPE Transformer](https://mlsanity.com/paper/2309.02612/) (2023)
- [Music2Latent: Consistency Autoencoders for Latent Audio Compression](https://mlsanity.com/paper/2408.06500/) (2024)
- [Conditioned-U-Net: Introducing a Control Mechanism in the U-Net for Multiple Source Separations](https://mlsanity.com/paper/1907.01277/) (2019)
- [D3Net: Densely connected multidilated DenseNet for music source separation](https://mlsanity.com/paper/2010.01733/) (2020)
- [Blind Audio Bandwidth Extension: A Diffusion-Based Zero-Shot Approach](https://mlsanity.com/paper/2306.01433/) (2023)
- [Generative Adversarial Source Separation](https://mlsanity.com/paper/1710.10779/) (2017)
- [End-to-end music source separation: is it possible in the waveform domain?](https://mlsanity.com/paper/1810.12187/) (2018)
- [Parallel and Flexible Sampling from Autoregressive Models via Langevin Dynamics](https://mlsanity.com/paper/2105.08164/) (2021)
- [Unsupervised Audio Source Separation using Generative Priors](https://mlsanity.com/paper/2005.13769/) (2020)
- [On loss functions and evaluation metrics for music source separation](https://mlsanity.com/paper/2202.07968/) (2022)

## Built on by (most cited first, 1 of 1)

- [ArrayDPS: Unsupervised Blind Speech Separation with a Diffusion Prior](https://mlsanity.com/paper/2505.05657/) (2025)

## Abstract (arXiv)

In this work, we propose an approach to music source separation that uses a generative diffusion model as a last-stage refinement on top of a deterministic separator, progressively enhancing the separated sources through iterative denoising. While the diffusion refinement yields measurable quality gains, it requires iterative steps at inference, increasing computational cost. To speed up the inference process, we apply consistency distillation, reducing inference to a single step while maintaining quality; with two or more steps, the distilled model even surpasses the diffusion-based approach. Crucially, our method is architecture-agnostic: we demonstrate state-of-the-art results when applied to both a custom U-Net-based separator on Slakh2100 and the state-of-the-art BS-RoFormer model on MUSDB18, showing that the refinement generalizes across backbone architectures. Sound examples are available at: https://consistency-separation.github.io/.