# Wave-U-Net: A Multi-Scale Neural Network for End-to-End Audio Source Separation

arXiv: https://arxiv.org/abs/1806.03185 · HTML: https://mlsanity.com/paper/1806.03185/

## Key facts

- Cited by 125 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Jun 2018.

## Built on by (most cited first, 30 of 125)

- [Hybrid Spectrogram and Waveform Source Separation](https://mlsanity.com/paper/2111.03600/) (2021)
- [Music Source Separation with Band-split RNN](https://mlsanity.com/paper/2209.15174/) (2022)
- [Mix and Localize: Localizing Sound Sources in Mixtures](https://mlsanity.com/paper/2211.15058/) (2022)
- [HiFi++: a Unified Framework for Bandwidth Extension and Speech Enhancement](https://mlsanity.com/paper/2203.13086/) (2022)
- [Separate Anything You Describe](https://mlsanity.com/paper/2308.05037/) (2023)
- [How to Design a Three-Stage Architecture for Audio-Visual Active Speaker Detection in the Wild](https://mlsanity.com/paper/2106.03932/) (2021)
- [FullSubNet+: Channel Attention FullSubNet with Complex Spectrograms for Speech Enhancement](https://mlsanity.com/paper/2203.12188/) (2022)
- [Zero-shot Audio Source Separation through Query-based Learning from Weakly-labeled Data](https://mlsanity.com/paper/2112.07891/) (2021)
- [Audio-Visual Speech Codecs: Rethinking Audio-Visual Speech Enhancement by Re-Synthesis](https://mlsanity.com/paper/2203.17263/) (2022)
- [A Generative Model for Raw Audio Using Transformer Architectures](https://mlsanity.com/paper/2106.16036/) (2021)
- [Avocodo: Generative Adversarial Network for Artifact-free Vocoder](https://mlsanity.com/paper/2206.13404/) (2022)
- [On loss functions and evaluation metrics for music source separation](https://mlsanity.com/paper/2202.07968/) (2022)
- [IAE-Net: Integral Autoencoders for Discretization-Invariant Learning](https://mlsanity.com/paper/2203.05142/) (2022)
- [Diffiner: A Versatile Diffusion-based Generative Refiner for Speech Enhancement](https://mlsanity.com/paper/2210.17287/) (2022)
- [Speech Denoising in the Waveform Domain with Self-Attention](https://mlsanity.com/paper/2202.07790/) (2022)
- [A cappella: Audio-visual Singing Voice Separation](https://mlsanity.com/paper/2104.09946/) (2021)
- [High Fidelity Speech Regeneration with Application to Speech Enhancement](https://mlsanity.com/paper/2102.00429/) (2021)
- [Sandglasset: A Light Multi-Granularity Self-attentive Network For Time-Domain Speech Separation](https://mlsanity.com/paper/2103.00819/) (2021)
- [Self-Attention for Audio Super-Resolution](https://mlsanity.com/paper/2108.11637/) (2021)
- [Wave-U-Mamba: An End-To-End Framework For High-Quality And Efficient Speech Super Resolution](https://mlsanity.com/paper/2409.09337/) (2024)
- [Effective Low-Cost Time-Domain Audio Separation Using Globally Attentive Locally Recurrent Networks](https://mlsanity.com/paper/2101.05014/) (2021)
- [Music Separation Enhancement with Generative Modeling](https://mlsanity.com/paper/2208.12387/) (2022)
- [A Generalized Bandsplit Neural Network for Cinematic Audio Source Separation](https://mlsanity.com/paper/2309.02539/) (2023)
- [HARP-Net: Hyper-Autoencoded Reconstruction Propagation for Scalable Neural Audio Coding](https://mlsanity.com/paper/2107.10843/) (2021)
- [Streamable Neural Audio Synthesis With Non-Causal Convolutions](https://mlsanity.com/paper/2204.07064/) (2022)
- [Exploiting Temporal Structures of Cyclostationary Signals for Data-Driven Single-Channel Source Separation](https://mlsanity.com/paper/2208.10325/) (2022)
- [A Stem-Agnostic Single-Decoder System for Music Source Separation Beyond Four Stems](https://mlsanity.com/paper/2406.18747/) (2024)
- [FINALLY: fast and universal speech enhancement with studio-like quality](https://mlsanity.com/paper/2410.05920/) (2024)
- [End-to-end Music Remastering System Using Self-supervised and Adversarial Training](https://mlsanity.com/paper/2202.08520/) (2022)
- [Benchmarks and leaderboards for sound demixing tasks](https://mlsanity.com/paper/2305.07489/) (2023)

## Abstract (arXiv)

Models for audio source separation usually operate on the magnitude spectrum, which ignores phase information and makes separation performance dependant on hyper-parameters for the spectral front-end. Therefore, we investigate end-to-end source separation in the time-domain, which allows modelling phase information and avoids fixed spectral transformations. Due to high sampling rates for audio, employing a long temporal input context on the sample level is difficult, but required for high quality separation results because of long-range temporal correlations. In this context, we propose the Wave-U-Net, an adaptation of the U-Net to the one-dimensional time domain, which repeatedly resamples feature maps to compute and combine features at different time scales. We introduce further architectural improvements, including an output layer that enforces source additivity, an upsampling technique and a context-aware prediction framework to reduce output artifacts. Experiments for singing voice separation indicate that our architecture yields a performance comparable to a state-of-the-art spectrogram-based U-Net architecture, given the same data. Finally, we reveal a problem with outliers in the currently used SDR evaluation metrics and suggest reporting rank-based statistics to alleviate this problem.