# RUMAA: Repeat-Aware Unified Music Audio Analysis for Score-Performance Alignment, Transcription, and Mistake Detection

arXiv: https://arxiv.org/abs/2507.12175 · HTML: https://mlsanity.com/paper/2507.12175/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 41 references that are in the map.
- First posted on arXiv 16 Jul 2025.

## Builds on (most cited first, 25 of 25)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Neural Turing Machines](https://mlsanity.com/paper/1410.5401/) (2014)
- [Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset](https://mlsanity.com/paper/1810.12247/) (2018)
- [LongNet: Scaling Transformers to 1,000,000,000 Tokens](https://mlsanity.com/paper/2307.02486/) (2023)
- [Compound Word Transformer: Learning to Compose Full-Song Music over Dynamic Directed Hypergraphs](https://mlsanity.com/paper/2101.02402/) (2021)
- [MT3: Multi-Task Multitrack Music Transcription](https://mlsanity.com/paper/2111.03017/) (2021)
- [Music2Latent: Consistency Autoencoders for Latent Audio Compression](https://mlsanity.com/paper/2408.06500/) (2024)
- [MIDI-DDSP: Detailed Control of Musical Performance via Hierarchical Modeling](https://mlsanity.com/paper/2112.09312/) (2021)
- [Automatic Piano Transcription with Hierarchical Frequency-Time Transformer](https://mlsanity.com/paper/2307.04305/) (2023)
- [Scoring Time Intervals using Non-Hierarchical Transformer For Automatic Piano Transcription](https://mlsanity.com/paper/2404.09466/) (2024)
- [YourMT3+: Multi-instrument Music Transcription with Enhanced Transformer Architectures and Cross-dataset Stem Augmentation](https://mlsanity.com/paper/2407.04822/) (2024)
- [CLaMP 2: Multimodal Music Information Retrieval Across 101 Languages Using Large Language Models](https://mlsanity.com/paper/2410.13267/) (2024)
- [TorchScale: Transformers at Scale](https://mlsanity.com/paper/2211.13184/) (2022)
- [Scaling Self-Supervised Representation Learning for Symbolic Piano Performance](https://mlsanity.com/paper/2506.23869/) (2025)
- [Audio-to-Score Alignment Using Deep Automatic Music Transcription](https://mlsanity.com/paper/2107.12854/) (2021)
- [Cross-Modal Music Retrieval and Applications: An Overview of Key Methodologies](https://mlsanity.com/paper/1902.04397/) (2019)
- [Music Performance Analysis: A Survey](https://mlsanity.com/paper/1907.00178/) (2019)
- [Structure-Aware Audio-to-Score Alignment using Progressively Dilated Convolutional Neural Networks](https://mlsanity.com/paper/2102.00382/) (2021)
- [PerTok: Expressive Encoding and Modeling of Symbolic Musical Ideas and Variations](https://mlsanity.com/paper/2410.02060/) (2024)
- [Improved Handling of Repeats and Jumps in Audio-Sheet Image Synchronization](https://mlsanity.com/paper/2007.14580/) (2020)
- [Detecting Music Performance Errors with Transformers](https://mlsanity.com/paper/2501.02030/) (2025)
- [ImprovNet -- Generating Controllable Musical Improvisations with Iterative Corruption Refinement](https://mlsanity.com/paper/2502.04522/) (2025)
- [TheGlueNote: Learned Representations for Robust and Flexible Note Alignment](https://mlsanity.com/paper/2408.04309/) (2024)
- [How does the teacher rate? Observations from the NeuroPiano dataset](https://mlsanity.com/paper/2410.03139/) (2024)

## Abstract (arXiv)

This study introduces RUMAA, a transformer-based framework for music performance analysis that unifies score-to-performance alignment, score-informed transcription, and mistake detection in a near end-to-end manner. Unlike prior methods addressing these tasks separately, RUMAA integrates them using pre-trained score and audio encoders and a novel tri-stream decoder capturing task interdependencies through proxy tasks. It aligns human-readable MusicXML scores with repeat symbols to full-length performance audio, overcoming traditional MIDI-based methods that rely on manually unfolded score-MIDI data with pre-specified repeat structures. RUMAA matches state-of-the-art alignment methods on non-repeated scores and outperforms them on scores with repeats in a public piano music dataset, while also delivering promising transcription and mistake detection results.