# End-to-end Piano Performance-MIDI to Score Conversion with Transformers

arXiv: https://arxiv.org/abs/2410.00210 · HTML: https://mlsanity.com/paper/2410.00210/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 47 references that are in the map.
- First posted on arXiv 30 Sep 2024.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://mlsanity.com/paper/2108.12409/) (2021)
- [Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks](https://mlsanity.com/paper/1506.03099/) (2015)
- [Onsets and Frames: Dual-Objective Piano Transcription](https://mlsanity.com/paper/1710.11153/) (2017)
- [MusicBERT: Symbolic Music Understanding with Large-Scale Pre-Training](https://mlsanity.com/paper/2106.05630/) (2021)
- [MT3: Multi-Task Multitrack Music Transcription](https://mlsanity.com/paper/2111.03017/) (2021)
- [Sequence-to-Sequence Piano Transcription with Transformers](https://mlsanity.com/paper/2107.09142/) (2021)
- [On the Potential of Simple Framewise Approaches to Piano Transcription](https://mlsanity.com/paper/1612.05153/) (2016)
- [GETMusic: Generating Any Music Tracks with a Unified Representation and Diffusion Framework](https://mlsanity.com/paper/2305.10841/) (2023)
- [Non-Local Musical Statistics as Guides for Audio-to-Score Piano Transcription](https://mlsanity.com/paper/2008.12710/) (2020)
- [PKSpell: Data-Driven Pitch Spelling and Key Signature Estimation](https://mlsanity.com/paper/2107.14009/) (2021)
- [Score Transformer: Generating Musical Score from Note-level Representation](https://mlsanity.com/paper/2112.00355/) (2021)
- [Evaluating Non-aligned Musical Score Transcriptions with MV2H](https://mlsanity.com/paper/1906.00566/) (2019)
- [Rhythm Transcription of Polyphonic Piano Music Based on Merged-Output HMM for Multiple Voices](https://mlsanity.com/paper/1701.08343/) (2017)
- [Note Value Recognition for Piano Transcription Using Markov Random Fields](https://mlsanity.com/paper/1703.08144/) (2017)

## Built on by (most cited first, 9 of 9)

- [Unified Cross-modal Translation of Score Images, Symbolic Music, and Performance Audio](https://mlsanity.com/paper/2505.12863/) (2025)
- [ABC-Eval: Benchmarking Large Language Models on Symbolic Music Understanding and Instruction Following](https://mlsanity.com/paper/2509.23350/) (2025)
- [LEGATO: Large-scale End-to-end Generalizable Approach to Typeset OMR](https://mlsanity.com/paper/2506.19065/) (2025)
- [Beat and Downbeat Tracking in Performance MIDI Using an End-to-End Transformer Architecture](https://mlsanity.com/paper/2507.00466/) (2025)
- [Beat-Based Rhythm Quantization of MIDI Performances](https://mlsanity.com/paper/2508.19262/) (2025)
- [EngravingGNN: A Hybrid Graph Neural Network for End-to-End Piano Score Engraving](https://mlsanity.com/paper/2509.19412/) (2025)
- [Disentangling Score Content and Performance Style for Joint Piano Rendering and Transcription](https://mlsanity.com/paper/2509.23878/) (2025)
- [Rubato: Transcribing Piano Music with Timestamps](https://mlsanity.com/paper/2605.24291/) (2026)
- [Audio-to-Score Transcription using Pre-trained Features, Data Augmentation, and the New SheetSage-A2S Dataset](https://mlsanity.com/paper/2608.06165/) (2026)

## Abstract (arXiv)

The automated creation of accurate musical notation from an expressive human performance is a fundamental task in computational musicology. To this end, we present an end-to-end deep learning approach that constructs detailed musical scores directly from real-world piano performance-MIDI files. We introduce a modern transformer-based architecture with a novel tokenized representation for symbolic music data. Framing the task as sequence-to-sequence translation rather than note-wise classification reduces alignment requirements and annotation costs, while allowing the prediction of more concise and accurate notation. To serialize symbolic music data, we design a custom tokenization stage based on compound tokens that carefully quantizes continuous values. This technique preserves more score information while reducing sequence lengths by 3.5× compared to prior approaches. Using the transformer backbone, our method demonstrates better understanding of note values, rhythmic structure, and details such as staff assignment. When evaluated end-to-end using transcription metrics such as MUSTER, we achieve significant improvements over previous deep learning approaches and complex HMM-based state-of-the-art pipelines. Our method is also the first to directly predict notational details like trill marks or stem direction from performance data. Code and models are available at https://github.com/TimFelixBeyer/MIDI2ScoreTransformer