# Simultaneous Separation and Transcription of Mixtures with Multiple Polyphonic and Percussive Instruments

arXiv: https://arxiv.org/abs/1910.12621 · HTML: https://mlsanity.com/paper/1910.12621/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Oct 2019.

## Built on by (most cited first, 18 of 18)

- [MT3: Multi-Task Multitrack Music Transcription](https://mlsanity.com/paper/2111.03017/) (2021)
- [Multi-instrument Music Synthesis with Spectrogram Diffusion](https://mlsanity.com/paper/2206.05408/) (2022)
- [Unaligned Supervision For Automatic Music Transcription in The Wild](https://mlsanity.com/paper/2204.13668/) (2022)
- [A Unified Model for Zero-shot Music Source Separation, Transcription and Synthesis](https://mlsanity.com/paper/2108.03456/) (2021)
- [The Chamber Ensemble Generator: Limitless High-Quality MIR Data via Generative Modeling](https://mlsanity.com/paper/2209.14458/) (2022)
- [Pop2Piano : Pop Audio-based Piano Cover Generation](https://mlsanity.com/paper/2211.00895/) (2022)
- [Music Separation Enhancement with Generative Modeling](https://mlsanity.com/paper/2208.12387/) (2022)
- [Jointist: Joint Learning for Multi-instrument Transcription and Its Applications](https://mlsanity.com/paper/2206.10805/) (2022)
- [Toward Fully Self-Supervised Multi-Pitch Estimation](https://mlsanity.com/paper/2402.15569/) (2024)
- [Jointist: Simultaneous Improvement of Multi-instrument Transcription and Music Source Separation via Joint Training](https://mlsanity.com/paper/2302.00286/) (2023)
- [The Inverse Drum Machine: Source Separation Through Joint Transcription and Analysis-by-Synthesis](https://mlsanity.com/paper/2505.03337/) (2025)
- [Unsupervised Source Separation By Steering Pretrained Music Models](https://mlsanity.com/paper/2110.13071/) (2021)
- [Analyzing and reducing the synthetic-to-real transfer gap in Music Information Retrieval: the task of automatic drum transcription](https://mlsanity.com/paper/2407.19823/) (2024)
- [Manifold-Aware Deep Clustering: Maximizing Angles between Embedding Vectors Based on Regular Simplex](https://mlsanity.com/paper/2106.02331/) (2021)
- [Pitch-Informed Instrument Assignment Using a Deep Convolutional Network with Multiple Kernel Shapes](https://mlsanity.com/paper/2107.13617/) (2021)
- [LightSAFT: Lightweight Latent Source Aware Frequency Transform for Source Separation](https://mlsanity.com/paper/2111.12516/) (2021)
- [Pre-trained Spatial Priors on Multichannel NMF for Music Source Separation](https://mlsanity.com/paper/2310.05821/) (2023)
- [Revisiting Input Time-frequency Representations in Multi-pitch Estimation for Vocal Ensembles](https://mlsanity.com/paper/2610.03656/) (2026)

## Abstract (arXiv)

We present a single deep learning architecture that can both separate an audio recording of a musical mixture into constituent single-instrument recordings and transcribe these instruments into a human-readable format at the same time, learning a shared musical representation for both tasks. This novel architecture, which we call Cerberus, builds on the Chimera network for source separation by adding a third "head" for transcription. By training each head with different losses, we are able to jointly learn how to separate and transcribe up to 5 instruments in our experiments with a single network. We show that the two tasks are highly complementary with one another and when learned jointly, lead to Cerberus networks that are better at both separation and transcription and generalize better to unseen mixtures.