# Interpreting Graphic Notation with MusicLDM: An AI Improvisation of Cornelius Cardew's Treatise

arXiv: https://arxiv.org/abs/2412.08944 · HTML: https://mlsanity.com/paper/2412.08944/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 20 references that are in the map.
- First posted on arXiv 12 Dec 2024.

## Builds on (most cited first, 17 of 17)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [DiffWave: A Versatile Diffusion Model for Audio Synthesis](https://mlsanity.com/paper/2009.09761/) (2020)
- [Jukebox: A Generative Model for Music](https://mlsanity.com/paper/2005.00341/) (2020)
- [MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation](https://mlsanity.com/paper/2302.08113/) (2023)
- [Simple and Controllable Music Generation](https://mlsanity.com/paper/2306.05284/) (2023)
- [MusicLM: Generating Music From Text](https://mlsanity.com/paper/2301.11325/) (2023)
- [AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining](https://mlsanity.com/paper/2308.05734/) (2023)
- [Adversarial Audio Synthesis](https://mlsanity.com/paper/1802.04208/) (2018)
- [Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset](https://mlsanity.com/paper/1810.12247/) (2018)
- [SampleRNN: An Unconditional End-to-End Neural Audio Generation Model](https://mlsanity.com/paper/1612.07837/) (2016)
- [Mustango: Toward Controllable Text-to-Music Generation](https://mlsanity.com/paper/2311.08355/) (2023)
- [Music ControlNet: Multiple Time-varying Controls for Music Generation](https://mlsanity.com/paper/2311.07069/) (2023)
- [DITTO: Diffusion Inference-Time T-Optimization for Music Generation](https://mlsanity.com/paper/2401.12179/) (2024)
- [Multi-instrument Music Synthesis with Spectrogram Diffusion](https://mlsanity.com/paper/2206.05408/) (2022)
- [Polyffusion: A Diffusion Model for Polyphonic Score Generation with Internal and External Controls](https://mlsanity.com/paper/2307.10304/) (2023)
- [Controllable Music Production with Diffusion Models and Guidance Gradients](https://mlsanity.com/paper/2311.00613/) (2023)

## Abstract (arXiv)

This work presents a novel method for composing and improvising music inspired by Cornelius Cardew's Treatise, using AI to bridge graphic notation and musical expression. By leveraging OpenAI's ChatGPT to interpret the abstract visual elements of Treatise, we convert these graphical images into descriptive textual prompts. These prompts are then input into MusicLDM, a pre-trained latent diffusion model designed for music generation. We introduce a technique called "outpainting," which overlaps sections of AI-generated music to create a seamless and cohesive composition. We demostrate a new perspective on performing and interpreting graphic scores, showing how AI can transform visual stimuli into sound and expand the creative possibilities in contemporary/experimental music composition. Musical pieces are available at https://bit.ly/TreatiseAI