# Melody transcription via generative pre-training

arXiv: https://arxiv.org/abs/2212.01884 · HTML: https://mlsanity.com/paper/2212.01884/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 65 references that are in the map.
- First posted on arXiv 4 Dec 2022.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Jukebox: A Generative Model for Music](https://mlsanity.com/paper/2005.00341/) (2020)
- [Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset](https://mlsanity.com/paper/1810.12247/) (2018)
- [Onsets and Frames: Dual-Objective Piano Transcription](https://mlsanity.com/paper/1710.11153/) (2017)
- [Cutting Music Source Separation Some Slakh: A Dataset to Study the Impact of Training Data Quality and Quantity](https://mlsanity.com/paper/1909.08494/) (2019)
- [MT3: Multi-Task Multitrack Music Transcription](https://mlsanity.com/paper/2111.03017/) (2021)
- [madmom: a new Python Audio and Music Signal Processing Library](https://mlsanity.com/paper/1605.07008/) (2016)
- [Codified audio language modeling learns useful representations for music information retrieval](https://mlsanity.com/paper/2107.05677/) (2021)
- [Learning Features of Music from Scratch](https://mlsanity.com/paper/1611.09827/) (2016)
- [Sequence-to-Sequence Piano Transcription with Transformers](https://mlsanity.com/paper/2107.09142/) (2021)
- [AI Song Contest: Human-AI Co-Creation in Songwriting](https://mlsanity.com/paper/2010.05388/) (2020)
- [Automatic Melody Harmonization with Triad Chords: A Comparative Study](https://mlsanity.com/paper/2001.02360/) (2020)
- [SurpriseNet: Melody Harmonization Conditioning on User-controlled Surprise Contours](https://mlsanity.com/paper/2108.00378/) (2021)
- [Melody Extraction from Polyphonic Music by Deep Learning Approaches: A Review](https://mlsanity.com/paper/2202.01078/) (2022)

## Built on by (most cited first, 24 of 24)

- [Do Music Generation Models Encode Music Theory?](https://mlsanity.com/paper/2410.00872/) (2024)
- [Music Foundation Model as Generic Booster for Music Downstream Tasks](https://mlsanity.com/paper/2411.01135/) (2024)
- [Emotion-driven Piano Music Generation via Two-stage Disentanglement and Functional Representation](https://mlsanity.com/paper/2407.20955/) (2024)
- [Emotion-Driven Melody Harmonization via Melodic Variation and Functional Representation](https://mlsanity.com/paper/2407.20176/) (2024)
- [Adaptive Accompaniment with ReaLchords](https://mlsanity.com/paper/2506.14723/) (2025)
- [Generative Adversarial Post-Training Mitigates Reward Hacking in Live Human-AI Music Interaction](https://mlsanity.com/paper/2511.17879/) (2025)
- [Scaling Up Music Information Retrieval Training with Semi-Supervised Learning](https://mlsanity.com/paper/2310.01353/) (2023)
- [Computational Copyright: Towards A Royalty Model for Music Generative AI](https://mlsanity.com/paper/2312.06646/) (2023)
- [Just Label the Repeats for In-The-Wild Audio-to-Score Alignment](https://mlsanity.com/paper/2411.07428/) (2024)
- [Music Plagiarism Detection: Problem Formulation and a Segment-based Solution](https://mlsanity.com/paper/2601.21260/) (2026)
- [Toward Leveraging Pre-Trained Self-Supervised Frontends for Automatic Singing Voice Understanding Tasks: Three Case Studies](https://mlsanity.com/paper/2306.12714/) (2023)
- [AMT-APC: Automatic Piano Cover by Fine-Tuning an Automatic Music Transcription Model](https://mlsanity.com/paper/2409.14086/) (2024)
- [Real-world Music Plagiarism Detection With Music Segment Transcription System](https://mlsanity.com/paper/2509.08282/) (2025)
- [Etude: Piano Cover Generation with a Three-Stage Approach -- Extract, strucTUralize, and DEcode](https://mlsanity.com/paper/2509.16522/) (2025)
- [MuseTok: Symbolic Music Tokenization for Generation and Semantic Understanding](https://mlsanity.com/paper/2510.16273/) (2025)
- [MIDI-LLM: Improving Text-to-MIDI Music Generation via Adapting Large Language Models](https://mlsanity.com/paper/2511.03942/) (2025)
- [Do Foundational Audio Encoders Understand Music Structure?](https://mlsanity.com/paper/2512.17209/) (2025)
- [What's a Credit Worth? A Market Framework for Attribution-Aware Compensation in Generative Music](https://mlsanity.com/paper/2607.00641/) (2026)
- [Dynamic HumTrans: Humming Transcription Using CNNs and Dynamic Programming](https://mlsanity.com/paper/2410.05455/) (2024)
- [Transcribing Rhythmic Patterns of the Guitar Track in Polyphonic Music](https://mlsanity.com/paper/2510.05756/) (2025)
- [S2Accompanist: A Semantic-Aware and Structure-Guided Diffusion Model for Music Accompaniment Generation](https://mlsanity.com/paper/2605.17414/) (2026)
- [MulTTiPop: A Multitrack Transcription Dataset for Pop Music](https://mlsanity.com/paper/2607.08756/) (2026)
- [Learning Music Style for Piano Arrangement Through Cross-Modal Bootstrapping](https://mlsanity.com/paper/2608.03050/) (2026)
- [Audio-to-Score Transcription using Pre-trained Features, Data Augmentation, and the New SheetSage-A2S Dataset](https://mlsanity.com/paper/2608.06165/) (2026)

## Abstract (arXiv)

Despite the central role that melody plays in music perception, it remains an open challenge in music information retrieval to reliably detect the notes of the melody present in an arbitrary music recording. A key challenge in melody transcription is building methods which can handle broad audio containing any number of instrument ensembles and musical styles - existing strategies work well for some melody instruments or styles but not all. To confront this challenge, we leverage representations from Jukebox (Dhariwal et al. 2020), a generative model of broad music audio, thereby improving performance on melody transcription by 20% relative to conventional spectrogram features. Another obstacle in melody transcription is a lack of training data - we derive a new dataset containing 50 hours of melody transcriptions from crowdsourced annotations of broad music. The combination of generative pre-training and a new dataset for this task results in 77% stronger performance on melody transcription relative to the strongest available baseline. By pairing our new melody transcription approach with solutions for beat detection, key estimation, and chord recognition, we build Sheet Sage, a system capable of transcribing human-readable lead sheets directly from music audio. Audio examples can be found at https://chrisdonahue.com/sheetsage and code at https://github.com/chrisdonahue/sheetsage .