# A Unified Model for Zero-shot Music Source Separation, Transcription and Synthesis

arXiv: https://arxiv.org/abs/2108.03456 · HTML: https://mlsanity.com/paper/2108.03456/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 30 references that are in the map.
- First posted on arXiv 7 Aug 2021.

## Builds on (most cited first, 17 of 17)

- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [The Sound of Pixels](https://mlsanity.com/paper/1804.03160/) (2018)
- [Deep clustering: Discriminative embeddings for segmentation and separation](https://mlsanity.com/paper/1508.04306/) (2015)
- [Music Gesture for Visual Sound Separation](https://mlsanity.com/paper/2004.09476/) (2020)
- [Creating A Multi-track Classical Musical Performance Dataset for Multimodal Music Analysis: Challenges, Insights, and Applications](https://mlsanity.com/paper/1612.08727/) (2016)
- [Conditioned-U-Net: Introducing a Control Mechanism in the U-Net for Multiple Source Separations](https://mlsanity.com/paper/1907.01277/) (2019)
- [MMDenseLSTM: An efficient combination of convolutional and recurrent neural networks for audio source separation](https://mlsanity.com/paper/1805.02410/) (2018)
- [Simultaneous Separation and Transcription of Mixtures with Multiple Polyphonic and Percussive Instruments](https://mlsanity.com/paper/1910.12621/) (2019)
- [Meta-learning Extractors for Music Source Separation](https://mlsanity.com/paper/2002.07016/) (2020)
- [Speaker-independent Speech Separation with Deep Attractor Network](https://mlsanity.com/paper/1707.03634/) (2017)
- [Audio query-based music source separation](https://mlsanity.com/paper/1908.06593/) (2019)
- [Class-conditional embeddings for music source separation](https://mlsanity.com/paper/1811.03076/) (2018)
- [Structured Dropout for Weak Label and Multi-Instance Learning and Its Application to Score-Informed Source Separation](https://mlsanity.com/paper/1609.04557/) (2016)
- [Multitask learning for instrument activation aware music source separation](https://mlsanity.com/paper/2008.00616/) (2020)
- [Exploring Aligned Lyrics-Informed Singing Voice Separation](https://mlsanity.com/paper/2008.04482/) (2020)
- [Fast Griffin Lim based Waveform Generation Strategy for Text-to-Speech Synthesis](https://mlsanity.com/paper/2007.05764/) (2020)
- [Content based singing voice source separation via strong conditioning using aligned phonemes](https://mlsanity.com/paper/2008.02070/) (2020)

## Built on by (most cited first, 18 of 18)

- [MT3: Multi-Task Multitrack Music Transcription](https://mlsanity.com/paper/2111.03017/) (2021)
- [Music Source Separation with Band-split RNN](https://mlsanity.com/paper/2209.15174/) (2022)
- [Multi-Source Diffusion Models for Simultaneous Music Generation and Separation](https://mlsanity.com/paper/2302.02257/) (2023)
- [Foundation Models for Music: A Survey](https://mlsanity.com/paper/2408.14340/) (2024)
- [Zero-shot Audio Source Separation through Query-based Learning from Weakly-labeled Data](https://mlsanity.com/paper/2112.07891/) (2021)
- [Structured Multi-Track Accompaniment Arrangement via Style Prior Modelling](https://mlsanity.com/paper/2310.16334/) (2023)
- [A Stem-Agnostic Single-Decoder System for Music Source Separation Beyond Four Stems](https://mlsanity.com/paper/2406.18747/) (2024)
- [Audio-to-symbolic Arrangement via Cross-modal Music Representation Learning](https://mlsanity.com/paper/2112.15110/) (2021)
- [Jointist: Joint Learning for Multi-instrument Transcription and Its Applications](https://mlsanity.com/paper/2206.10805/) (2022)
- [Jointist: Simultaneous Improvement of Multi-instrument Transcription and Music Source Separation via Joint Training](https://mlsanity.com/paper/2302.00286/) (2023)
- [DisMix: Disentangling Mixtures of Musical Instruments for Source-level Pitch and Timbre Manipulation](https://mlsanity.com/paper/2408.10807/) (2024)
- [Unsupervised Disentanglement of Content and Style via Variance-Invariance Constraints](https://mlsanity.com/paper/2407.03824/) (2024)
- [Self-refining of Pseudo Labels for Music Source Separation with Noisy Labeled Data](https://mlsanity.com/paper/2307.12576/) (2023)
- [LightSAFT: Lightweight Latent Source Aware Frequency Transform for Source Separation](https://mlsanity.com/paper/2111.12516/) (2021)
- [MAJL: A Model-Agnostic Joint Learning Framework for Music Source Separation and Pitch Estimation](https://mlsanity.com/paper/2501.03689/) (2025)
- [A Conditioned UNet for Music Source Separation](https://mlsanity.com/paper/2512.15532/) (2025)
- [The Costs of Reproducibility in Music Separation Research: a Replication of Band-Split RNN](https://mlsanity.com/paper/2603.09187/) (2026)
- [DDSynth-RL: Audio Synthesizer Inversion via Discrete Diffusion with Reinforcement Learning](https://mlsanity.com/paper/2608.03032/) (2026)

## Abstract (arXiv)

We propose a unified model for three inter-related tasks: 1) to textitseparate individual sound sources from a mixed music audio, 2) to textittranscribe each sound source to MIDI notes, and 3) totextit synthesize new pieces based on the timbre of separated sources. The model is inspired by the fact that when humans listen to music, our minds can not only separate the sounds of different instruments, but also at the same time perceive high-level representations such as score and timbre. To mirror such capability computationally, we designed a pitch-timbre disentanglement module based on a popular encoder-decoder neural architecture for source separation. The key inductive biases are vector-quantization for pitch representation and pitch-transformation invariant for timbre representation. In addition, we adopted a query-by-example method to achieve textitzero-shot learning, i.e., the model is capable of doing source separation, transcription, and synthesis for textitunseen instruments. The current design focuses on audio mixtures of two monophonic instruments. Experimental results show that our model outperforms existing multi-task baselines, and the transcribed score serves as a powerful auxiliary for separation tasks.