# Hybrid Spectrogram and Waveform Source Separation

arXiv: https://arxiv.org/abs/2111.03600 · HTML: https://mlsanity.com/paper/2111.03600/

## Key facts

- Cited by 91 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 34 references that are in the map.
- First posted on arXiv 5 Nov 2021.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Multi-Scale Context Aggregation by Dilated Convolutions](https://mlsanity.com/paper/1511.07122/) (2015)
- [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://mlsanity.com/paper/2108.12409/) (2021)
- [Going deeper with Image Transformers](https://mlsanity.com/paper/2103.17239/) (2021)
- [Language Modeling with Gated Convolutional Networks](https://mlsanity.com/paper/1612.08083/) (2016)
- [Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation](https://mlsanity.com/paper/1809.07454/) (2018)
- [MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis](https://mlsanity.com/paper/1910.06711/) (2019)
- [Spectral Norm Regularization for Improving the Generalizability of Deep Learning](https://mlsanity.com/paper/1705.10941/) (2017)
- [Dual-path RNN: efficient long sequence modeling for time-domain single-channel speech separation](https://mlsanity.com/paper/1910.06379/) (2019)
- [Wave-U-Net: A Multi-Scale Neural Network for End-to-End Audio Source Separation](https://mlsanity.com/paper/1806.03185/) (2018)
- [Music Source Separation in the Waveform Domain](https://mlsanity.com/paper/1911.13254/) (2019)
- [Decoupling Magnitude and Phase Estimation with Deep ResUNet for Music Source Separation](https://mlsanity.com/paper/2109.05418/) (2021)
- [The 2018 Signal Separation Evaluation Campaign](https://mlsanity.com/paper/1804.06267/) (2018)
- [Music Demixing Challenge 2021](https://mlsanity.com/paper/2108.13559/) (2021)
- [Upsampling artifacts in neural audio synthesis](https://mlsanity.com/paper/2010.14356/) (2020)
- [D3Net: Densely connected multidilated DenseNet for music source separation](https://mlsanity.com/paper/2010.01733/) (2020)
- [PoCoNet: Better Speech Enhancement with Frequency-Positional Embeddings, Semi-Supervised Conversational Data, and Biased Loss](https://mlsanity.com/paper/2008.04470/) (2020)
- [All for One and One for All: Improving Music Separation by Bridging Networks](https://mlsanity.com/paper/2010.04228/) (2020)
- [End-to-end music source separation: is it possible in the waveform domain?](https://mlsanity.com/paper/1810.12187/) (2018)
- [LaSAFT: Latent Source Attentive Frequency Transformation for Conditioned Source Separation](https://mlsanity.com/paper/2010.11631/) (2020)

## Built on by (most cited first, 30 of 91)

- [Music Source Separation with Band-split RNN](https://mlsanity.com/paper/2209.15174/) (2022)
- [Multi-Source Diffusion Models for Simultaneous Music Generation and Separation](https://mlsanity.com/paper/2302.02257/) (2023)
- [LeVo: High-Quality Song Generation with Multi-Preference Alignment](https://mlsanity.com/paper/2506.07520/) (2025)
- [UniMuMo: Unified Text, Music and Motion Generation](https://mlsanity.com/paper/2410.04534/) (2024)
- [Music Mixing Style Transfer: A Contrastive Learning Approach to Disentangle Audio Effects](https://mlsanity.com/paper/2211.02247/) (2022)
- [AERO: Audio Super Resolution in the Spectral Domain](https://mlsanity.com/paper/2211.12232/) (2022)
- [JEN-1 Composer: A Unified Framework for High-Fidelity Multi-Track Music Generation](https://mlsanity.com/paper/2310.19180/) (2023)
- [MuChin: A Chinese Colloquial Description Benchmark for Evaluating Language Models in the Field of Music](https://mlsanity.com/paper/2402.09871/) (2024)
- [IndicVoices-R: Unlocking a Massive Multilingual Multi-speaker Speech Corpus for Scaling Indian TTS](https://mlsanity.com/paper/2409.05356/) (2024)
- [Analyzable Chain-of-Musical-Thought Prompting for High-Fidelity Music Generation](https://mlsanity.com/paper/2503.19611/) (2025)
- [Simultaneous Music Separation and Generation Using Multi-Track Latent Diffusion Models](https://mlsanity.com/paper/2409.12346/) (2024)
- [Latent Autoregressive Source Separation](https://mlsanity.com/paper/2301.08562/) (2023)
- [SongCreator: Lyrics-based Universal Song Generation](https://mlsanity.com/paper/2409.06029/) (2024)
- [Songs Across Borders: Singable and Controllable Neural Lyric Translation](https://mlsanity.com/paper/2305.16816/) (2023)
- [MusiConGen: Rhythm and Chord Control for Transformer-Based Text-to-Music Generation](https://mlsanity.com/paper/2407.15060/) (2024)
- [JAM: A Tiny Flow-based Song Generator with Fine-grained Controllability and Aesthetic Alignment](https://mlsanity.com/paper/2507.20880/) (2025)
- [Music Separation Enhancement with Generative Modeling](https://mlsanity.com/paper/2208.12387/) (2022)
- [A Generalized Bandsplit Neural Network for Cinematic Audio Source Separation](https://mlsanity.com/paper/2309.02539/) (2023)
- [Evaluating Self-supervised Speech Models on a Taiwanese Hokkien Corpus](https://mlsanity.com/paper/2312.06668/) (2023)
- [A Stem-Agnostic Single-Decoder System for Music Source Separation Beyond Four Stems](https://mlsanity.com/paper/2406.18747/) (2024)
- [Implicit Neural Spatial Filtering for Multichannel Source Separation in the Waveform Domain](https://mlsanity.com/paper/2206.15423/) (2022)
- [Benchmarks and leaderboards for sound demixing tasks](https://mlsanity.com/paper/2305.07489/) (2023)
- [SonicMaster: Towards Controllable All-in-One Music Restoration and Mastering](https://mlsanity.com/paper/2508.03448/) (2025)
- [UnityShots: Memory-Driven Multi-Shot Audio-Video Generation with Boundary-Aware Gating](https://mlsanity.com/paper/2606.21661/) (2026)
- [JaCappella Corpus: A Japanese a Cappella Vocal Ensemble Corpus](https://mlsanity.com/paper/2211.16028/) (2022)
- [Recognizing Ornaments in Vocal Indian Art Music with Active Annotation](https://mlsanity.com/paper/2505.04419/) (2025)
- [Learning to Highlight Audio by Watching Movies](https://mlsanity.com/paper/2505.12154/) (2025)
- [Music Source Restoration](https://mlsanity.com/paper/2505.21827/) (2025)
- [Distortion Audio Effects: Learning How to Recover the Clean Signal](https://mlsanity.com/paper/2202.01664/) (2022)
- [A differentiable short-time Fourier transform with respect to the window length](https://mlsanity.com/paper/2208.10886/) (2022)

## Abstract (arXiv)

Source separation models either work on the spectrogram or waveform domain. In this work, we show how to perform end-to-end hybrid source separation, letting the model decide which domain is best suited for each source, and even combining both. The proposed hybrid version of the Demucs architecture won the Music Demixing Challenge 2021 organized by Sony. This architecture also comes with additional improvements, such as compressed residual branches, local attention or singular value regularization. Overall, a 1.4 dB improvement of the Signal-To-Distortion (SDR) was observed across all sources as measured on the MusDB HQ dataset, an improvement confirmed by human subjective evaluation, with an overall quality rated at 2.83 out of 5 (2.36 for the non hybrid Demucs), and absence of contamination at 3.04 (against 2.37 for the non hybrid Demucs and 2.44 for the second ranking model submitted at the competition).