# The Interpretation Gap in Text-to-Music Generation Models

arXiv: https://arxiv.org/abs/2407.10328 · HTML: https://mlsanity.com/paper/2407.10328/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 56 references that are in the map.
- First posted on arXiv 14 Jul 2024.

## Builds on (most cited first, 30 of 31)

- [WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research](https://mlsanity.com/paper/2303.17395/) (2023)
- [Fast Timing-Conditioned Latent Audio Diffusion](https://mlsanity.com/paper/2402.04825/) (2024)
- [Mustango: Toward Controllable Text-to-Music Generation](https://mlsanity.com/paper/2311.08355/) (2023)
- [Tango 2: Aligning Diffusion-based Text-to-Audio Generations through Direct Preference Optimization](https://mlsanity.com/paper/2404.09956/) (2024)
- [Long-form music generation with latent diffusion](https://mlsanity.com/paper/2404.10301/) (2024)
- [LP-MusicCaps: LLM-Based Pseudo Music Captioning](https://mlsanity.com/paper/2307.16372/) (2023)
- [Music ControlNet: Multiple Time-varying Controls for Music Generation](https://mlsanity.com/paper/2311.07069/) (2023)
- [Music Understanding LLaMA: Advancing Text-to-Music Generation with Question Answering and Captioning](https://mlsanity.com/paper/2308.11276/) (2023)
- [ChatMusician: Understanding and Generating Music Intrinsically with LLM](https://mlsanity.com/paper/2402.16153/) (2024)
- [Learning Interpretable Representation for Controllable Polyphonic Music Generation](https://mlsanity.com/paper/2008.07122/) (2020)
- [Diff-A-Riff: Musical Accompaniment Co-creation via Latent Diffusion Models](https://mlsanity.com/paper/2406.08384/) (2024)
- [MusicMagus: Zero-Shot Text-to-Music Editing via Diffusion Models](https://mlsanity.com/paper/2402.06178/) (2024)
- [Joint Audio and Symbolic Conditioning for Temporally Controlled Text-to-Music Generation](https://mlsanity.com/paper/2406.10970/) (2024)
- [Audio-Text Models Do Not Yet Leverage Natural Language](https://mlsanity.com/paper/2303.10667/) (2023)
- [AI Song Contest: Human-AI Co-Creation in Songwriting](https://mlsanity.com/paper/2010.05388/) (2020)
- [A Differentiable Perceptual Audio Metric Learned from Just Noticeable Differences](https://mlsanity.com/paper/2001.04460/) (2020)
- [InstructME: An Instruction Guided Music Edit And Remix Framework with Latent Diffusion Models](https://mlsanity.com/paper/2308.14360/) (2023)
- [Learning Disentangled Representations of Timbre and Pitch for Musical Instrument Sounds Using Gaussian Mixture Variational Autoencoders](https://mlsanity.com/paper/1906.08152/) (2019)
- [Content-based Controls For Music Large Language Modeling](https://mlsanity.com/paper/2310.17162/) (2023)
- [T-CLAP: Temporal-Enhanced Contrastive Language-Audio Pretraining](https://mlsanity.com/paper/2404.17806/) (2024)
- [JEN-1 Composer: A Unified Framework for High-Fidelity Multi-Track Music Generation](https://mlsanity.com/paper/2310.19180/) (2023)
- [Arrange, Inpaint, and Refine: Steerable Long-term Music Audio Generation and Editing via Content-based Controls](https://mlsanity.com/paper/2402.09508/) (2024)
- [Loop Copilot: Conducting AI Ensembles for Music Generation and Iterative Editing](https://mlsanity.com/paper/2310.12404/) (2023)
- [Instruct-MusicGen: Unlocking Text-to-Music Editing for Music Language Models via Instruction Tuning](https://mlsanity.com/paper/2405.18386/) (2024)
- [Generalized Multi-Source Inference for Text Conditioned Music Diffusion Models](https://mlsanity.com/paper/2403.11706/) (2024)
- [Perceptual Loss Function for Neural Modelling of Audio Systems](https://mlsanity.com/paper/1911.08922/) (2019)
- [Listen, Chat, and Remix: Text-Guided Soundscape Remixing for Enhanced Auditory Experience](https://mlsanity.com/paper/2402.03710/) (2024)
- [PAGURI: a user experience study of creative interaction with text-to-music models](https://mlsanity.com/paper/2407.04333/) (2024)
- [IteraTTA: An interface for exploring both text prompts and audio priors in generating music with text-to-audio models](https://mlsanity.com/paper/2307.13005/) (2023)
- [Moving to Communicate, Moving to Interact: Patterns of Body Motion in Musical Duo Performance](https://mlsanity.com/paper/1911.09018/) (2019)

## Built on by (most cited first, 1 of 1)

- [AI-Assisted Music Production: A User Study on Text-to-Music Models](https://mlsanity.com/paper/2509.23364/) (2025)

## Abstract (arXiv)

Large-scale text-to-music generation models have significantly enhanced music creation capabilities, offering unprecedented creative freedom. However, their ability to collaborate effectively with human musicians remains limited. In this paper, we propose a framework to describe the musical interaction process, which includes expression, interpretation, and execution of controls. Following this framework, we argue that the primary gap between existing text-to-music models and musicians lies in the interpretation stage, where models lack the ability to interpret controls from musicians. We also propose two strategies to address this gap and call on the music information retrieval community to tackle the interpretation challenge to improve human-AI musical collaboration.