# Audio-to-symbolic Arrangement via Cross-modal Music Representation Learning

arXiv: https://arxiv.org/abs/2112.15110 · HTML: https://mlsanity.com/paper/2112.15110/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 23 references that are in the map.
- First posted on arXiv 30 Dec 2021.

## Builds on (most cited first, 15 of 15)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset](https://mlsanity.com/paper/1810.12247/) (2018)
- [DDSP: Differentiable Digital Signal Processing](https://mlsanity.com/paper/2001.04643/) (2020)
- [Onsets and Frames: Dual-Objective Piano Transcription](https://mlsanity.com/paper/1710.11153/) (2017)
- [POP909: A Pop-song Dataset for Music Arrangement Generation](https://mlsanity.com/paper/2008.07142/) (2020)
- [Deep Learning Techniques for Music Generation -- A Survey](https://mlsanity.com/paper/1709.01620/) (2017)
- [Learning Interpretable Representation for Controllable Polyphonic Music Generation](https://mlsanity.com/paper/2008.07122/) (2020)
- [Deep Music Analogy Via Latent Representation Disentanglement](https://mlsanity.com/paper/1906.03626/) (2019)
- [Music SketchNet: Controllable Music Generation via Factorized Representations of Pitch and Rhythm](https://mlsanity.com/paper/2008.01291/) (2020)
- [PIANOTREE VAE: Structured Representation Learning for Polyphonic Music](https://mlsanity.com/paper/2008.07118/) (2020)
- [A Unified Model for Zero-shot Music Source Separation, Transcription and Synthesis](https://mlsanity.com/paper/2108.03456/) (2021)
- [Music Style Transfer: A Position Paper](https://mlsanity.com/paper/1803.06841/) (2018)
- [PerformanceNet: Score-to-Audio Music Generation with Multi-Band Convolutional Residual Network](https://mlsanity.com/paper/1811.04357/) (2018)
- [Musical Composition Style Transfer via Disentangled Timbre Representations](https://mlsanity.com/paper/1905.13567/) (2019)
- [Learning a Latent Space of Multitrack Measures](https://mlsanity.com/paper/1806.00195/) (2018)

## Built on by (most cited first, 6 of 6)

- [Whole-Song Hierarchical Generation of Symbolic Music Using Cascaded Diffusion Models](https://mlsanity.com/paper/2405.09901/) (2024)
- [Structured Multi-Track Accompaniment Arrangement via Style Prior Modelling](https://mlsanity.com/paper/2310.16334/) (2023)
- [Unified Cross-modal Translation of Score Images, Symbolic Music, and Performance Audio](https://mlsanity.com/paper/2505.12863/) (2025)
- [Unifying Symbolic Music Arrangement: Track-Aware Reconstruction and Structured Tokenization](https://mlsanity.com/paper/2408.15176/) (2024)
- [Unsupervised Disentanglement of Content and Style via Variance-Invariance Constraints](https://mlsanity.com/paper/2407.03824/) (2024)
- [Learning Music Style for Piano Arrangement Through Cross-Modal Bootstrapping](https://mlsanity.com/paper/2608.03050/) (2026)

## Abstract (arXiv)

Could we automatically derive the score of a piano accompaniment based on the audio of a pop song? This is the audio-to-symbolic arrangement problem we tackle in this paper. A good arrangement model should not only consider the audio content but also have prior knowledge of piano composition (so that the generation "sounds like" the audio and meanwhile maintains musicality). To this end, we contribute a cross-modal representation-learning model, which 1) extracts chord and melodic information from the audio, and 2) learns texture representation from both audio and a corrupted ground truth arrangement. We further introduce a tailored training strategy that gradually shifts the source of texture information from corrupted score to audio. In the end, the score-based texture posterior is reduced to a standard normal distribution, and only audio is needed for inference. Experiments show that our model captures major audio information and outperforms baselines in generation quality.