# Libretto: Giving LLM Agents a Sense of Musical Structure

arXiv: https://arxiv.org/abs/2606.22708 · HTML: https://mlsanity.com/paper/2606.22708/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use.
- Builds on 14 of its 24 references that are in the map.
- First posted on arXiv 21 Jun 2026.

## Placed in

- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 14 of 14)

- [Simple and Controllable Music Generation](https://mlsanity.com/paper/2306.05284/) (2023)
- [MuseGAN: Multi-track Sequential Generative Adversarial Networks for Symbolic Music Generation and Accompaniment](https://mlsanity.com/paper/1709.06298/) (2017)
- [Meta Audiobox Aesthetics: Unified Automatic Quality Assessment for Speech, Music, and Sound](https://mlsanity.com/paper/2502.05139/) (2025)
- [A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music](https://mlsanity.com/paper/1803.05428/) (2018)
- [DeepBach: a Steerable Model for Bach Chorales Generation](https://mlsanity.com/paper/1612.01010/) (2016)
- [ChatMusician: Understanding and Generating Music Intrinsically with LLM](https://mlsanity.com/paper/2402.16153/) (2024)
- [Museformer: Transformer with Fine- and Coarse-Grained Attention for Music Generation](https://mlsanity.com/paper/2210.10349/) (2022)
- [Anticipatory Music Transformer](https://mlsanity.com/paper/2306.08620/) (2023)
- [Symphony Generation with Permutation Invariant Language Model](https://mlsanity.com/paper/2205.05448/) (2022)
- [MuPT: A Generative Symbolic Music Pretrained Transformer](https://mlsanity.com/paper/2404.06393/) (2024)
- [NotaGen: Advancing Musicality in Symbolic Music Generation with Large Language Model Training Paradigms](https://mlsanity.com/paper/2502.18008/) (2025)
- [Generating Symbolic Music from Natural Language Prompts using an LLM-Enhanced Dataset](https://mlsanity.com/paper/2410.02084/) (2024)
- [ABC-Eval: Benchmarking Large Language Models on Symbolic Music Understanding and Instruction Following](https://mlsanity.com/paper/2509.23350/) (2025)
- [MIDI-LLM: Improving Text-to-MIDI Music Generation via Adapting Large Language Models](https://mlsanity.com/paper/2511.03942/) (2025)

## Abstract (arXiv)

Generative music systems can now produce impressive audio from text prompts, but audio outputs are difficult to inspect, edit, and diagnose as musical structure. We introduce Libretto, an agent-facing framework for symbolic music generation and revision. Libretto uses an LLM-native grammar with explicit onset slots, voices, and bar-level organization, then evaluates each piece in a corpus-calibrated statistical space over rhythm, harmony, melody, texture, form, and variation. The same structural axes support retrieval, diagnosis, copy-risk control, and iterative self-revision. Across gap filling, reference-guided full-piece generation, gradual morphing, and educational music generation, Libretto turns symbolic music from a raw token sequence into a measurable and editable object for language-model agents.