# MR-MT3: Memory Retaining Multi-Track Music Transcription to Mitigate Instrument Leakage

arXiv: https://arxiv.org/abs/2403.10024 · HTML: https://mlsanity.com/paper/2403.10024/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 27 references that are in the map.
- First posted on arXiv 15 Mar 2024.

## Builds on (most cited first, 17 of 17)

- [Pix2seq: A Language Modeling Framework for Object Detection](https://mlsanity.com/paper/2109.10852/) (2021)
- [Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders](https://mlsanity.com/paper/1704.01279/) (2017)
- [MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognition](https://mlsanity.com/paper/2201.08383/) (2022)
- [Onsets and Frames: Dual-Objective Piano Transcription](https://mlsanity.com/paper/1710.11153/) (2017)
- [MART: Memory-Augmented Recurrent Transformer for Coherent Video Paragraph Captioning](https://mlsanity.com/paper/2005.05402/) (2020)
- [MT3: Multi-Task Multitrack Music Transcription](https://mlsanity.com/paper/2111.03017/) (2021)
- [Learning Features of Music from Scratch](https://mlsanity.com/paper/1611.09827/) (2016)
- [Sequence-to-Sequence Piano Transcription with Transformers](https://mlsanity.com/paper/2107.09142/) (2021)
- [Multi-instrument Music Synthesis with Spectrogram Diffusion](https://mlsanity.com/paper/2206.05408/) (2022)
- [A Lightweight Instrument-Agnostic Model for Polyphonic Note Transcription and Multipitch Estimation](https://mlsanity.com/paper/2203.09893/) (2022)
- [ReconVAT: A Semi-Supervised Automatic Music Transcription Framework for Low-Resource Real-World Data](https://mlsanity.com/paper/2107.04954/) (2021)
- [Deep Polyphonic ADSR Piano Note Transcription](https://mlsanity.com/paper/1906.09165/) (2019)
- [DiffRoll: Diffusion-based Generative Music Transcription with Unsupervised Pretraining Capability](https://mlsanity.com/paper/2210.05148/) (2022)
- [Multitrack Music Transcription with a Time-Frequency Perceiver](https://mlsanity.com/paper/2306.10785/) (2023)
- [ComMU: Dataset for Combinatorial Music Generation](https://mlsanity.com/paper/2211.09385/) (2022)
- [Jointist: Simultaneous Improvement of Multi-instrument Transcription and Music Source Separation via Joint Training](https://mlsanity.com/paper/2302.00286/) (2023)
- [Timbre-Trap: A Low-Resource Framework for Instrument-Agnostic Music Transcription](https://mlsanity.com/paper/2309.15717/) (2023)

## Built on by (most cited first, 2 of 2)

- [LadderSym: A Multimodal Interleaved Transformer for Music Practice Error Detection](https://mlsanity.com/paper/2510.08580/) (2025)
- [Playability-Aware Audio-to-Tablature Guitar Transcription via Diffusion Models](https://mlsanity.com/paper/2608.30854/) (2026)

## Abstract (arXiv)

This paper presents enhancements to the MT3 model, a state-of-the-art (SOTA) token-based multi-instrument automatic music transcription (AMT) model. Despite SOTA performance, MT3 has the issue of instrument leakage, where transcriptions are fragmented across different instruments. To mitigate this, we propose MR-MT3, with enhancements including a memory retention mechanism, prior token sampling, and token shuffling are proposed. These methods are evaluated on the Slakh2100 dataset, demonstrating improved onset F1 scores and reduced instrument leakage. In addition to the conventional multi-instrument transcription F1 score, new metrics such as the instrument leakage ratio and the instrument detection F1 score are introduced for a more comprehensive assessment of transcription quality. The study also explores the issue of domain overfitting by evaluating MT3 on single-instrument monophonic datasets such as ComMU and NSynth. The findings, along with the source code, are shared to facilitate future work aimed at refining token-based multi-instrument AMT models.