# Towards Musically Informed Evaluation of Piano Transcription Models

arXiv: https://arxiv.org/abs/2406.08454 · HTML: https://mlsanity.com/paper/2406.08454/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Jun 2024.

## Built on by (most cited first, 5 of 5)

- [Quantifying the Corpus Bias Problem in Automatic Music Transcription Systems](https://mlsanity.com/paper/2408.04737/) (2024)
- [Sound and Music Biases in Deep Music Transcription Models: A Systematic Analysis](https://mlsanity.com/paper/2512.14602/) (2025)
- [Pay Attention to the Keys: Visual Piano Transcription Using Transformers](https://mlsanity.com/paper/2411.09037/) (2024)
- [PianoCoRe: Combined and Refined Piano MIDI Dataset](https://mlsanity.com/paper/2605.06627/) (2026)
- [Learning to Predict Performance-induced Emotion Differences in Classical Piano Music](https://mlsanity.com/paper/2607.28876/) (2026)

## Abstract (arXiv)

Automatic piano transcription models are typically evaluated using simple frame- or note-wise information retrieval (IR) metrics. Such benchmark metrics do not provide insights into the transcription quality of specific musical aspects such as articulation, dynamics, or rhythmic precision of the output, which are essential in the context of expressive performance analysis. Furthermore, in recent years, MAESTRO has become the de-facto training and evaluation dataset for such models. However, inference performance has been observed to deteriorate substantially when applied on out-of-distribution data, thereby questioning the suitability and reliability of transcribed outputs from such models for specific MIR tasks. In this work, we investigate the performance of three state-of-the-art piano transcription models in two experiments. In the first one, we propose a variety of musically informed evaluation metrics which, in contrast to the IR metrics, offer more detailed insight into the musical quality of the transcriptions. In the second experiment, we compare inference performance on real-world and perturbed audio recordings, and highlight musical dimensions which our metrics can help explain. Our experimental results highlight the weaknesses of existing piano transcription metrics and contribute to a more musically sound error analysis of transcription outputs.