# Timbre-Trap: A Low-Resource Framework for Instrument-Agnostic Music Transcription

arXiv: https://arxiv.org/abs/2309.15717 · HTML: https://mlsanity.com/paper/2309.15717/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 35 references that are in the map.
- First posted on arXiv 27 Sep 2023.

## Builds on (most cited first, 14 of 14)

- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [SoundStream: An End-to-End Neural Audio Codec](https://mlsanity.com/paper/2107.03312/) (2021)
- [Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset](https://mlsanity.com/paper/1810.12247/) (2018)
- [FMA: A Dataset For Music Analysis](https://mlsanity.com/paper/1612.01840/) (2016)
- [DDSP: Differentiable Digital Signal Processing](https://mlsanity.com/paper/2001.04643/) (2020)
- [Onsets and Frames: Dual-Objective Piano Transcription](https://mlsanity.com/paper/1710.11153/) (2017)
- [MT3: Multi-Task Multitrack Music Transcription](https://mlsanity.com/paper/2111.03017/) (2021)
- [Solving Audio Inverse Problems with a Diffusion Model](https://mlsanity.com/paper/2210.15228/) (2022)
- [Creating A Multi-track Classical Musical Performance Dataset for Multimodal Music Analysis: Challenges, Insights, and Applications](https://mlsanity.com/paper/1612.08727/) (2016)
- [A Lightweight Instrument-Agnostic Model for Polyphonic Note Transcription and Multipitch Estimation](https://mlsanity.com/paper/2203.09893/) (2022)
- [ReconVAT: A Semi-Supervised Automatic Music Transcription Framework for Low-Resource Real-World Data](https://mlsanity.com/paper/2107.04954/) (2021)
- [Cross-domain Neural Pitch and Periodicity Estimation](https://mlsanity.com/paper/2301.12258/) (2023)
- [Multitrack Music Transcription with a Time-Frequency Perceiver](https://mlsanity.com/paper/2306.10785/) (2023)
- [The Effect of Spectrogram Reconstruction on Automatic Music Transcription: An Alternative Approach to Improve Transcription Accuracy](https://mlsanity.com/paper/2010.09969/) (2020)

## Built on by (most cited first, 3 of 3)

- [Toward Fully Self-Supervised Multi-Pitch Estimation](https://mlsanity.com/paper/2402.15569/) (2024)
- [MR-MT3: Memory Retaining Multi-Track Music Transcription to Mitigate Instrument Leakage](https://mlsanity.com/paper/2403.10024/) (2024)
- [Investigating an Overfitting and Degeneration Phenomenon in Self-Supervised Multi-Pitch Estimation](https://mlsanity.com/paper/2506.23371/) (2025)

## Abstract (arXiv)

In recent years, research on music transcription has focused mainly on architecture design and instrument-specific data acquisition. With the lack of availability of diverse datasets, progress is often limited to solo-instrument tasks such as piano transcription. Several works have explored multi-instrument transcription as a means to bolster the performance of models on low-resource tasks, but these methods face the same data availability issues. We propose Timbre-Trap, a novel framework which unifies music transcription and audio reconstruction by exploiting the strong separability between pitch and timbre. We train a single autoencoder to simultaneously estimate pitch salience and reconstruct complex spectral coefficients, selecting between either output during the decoding stage via a simple switch mechanism. In this way, the model learns to produce coefficients corresponding to timbre-less audio, which can be interpreted as pitch salience. We demonstrate that the framework leads to performance comparable to state-of-the-art instrument-agnostic transcription methods, while only requiring a small amount of annotated data.