The Interpretation Gap in Text-to-Music Generation Models
Not on the map. Why?
Abstract (arXiv)
Large-scale text-to-music generation models have significantly enhanced music creation capabilities, offering unprecedented creative freedom. However, their ability to collaborate effectively with human musicians remains limited. In this paper, we propose a framework to describe the musical interaction process, which includes expression, interpretation, and execution of controls. Following this framework, we argue that the primary gap between existing text-to-music models and musicians lies in the interpretation stage, where models lack the ability to interpret controls from musicians. We also propose two strategies to address this gap and call on the music information retrieval community to tackle the interpretation challenge to improve human-AI musical collaboration.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 31 of its 56 references that are in the map. First posted on arXiv 14 Jul 2024.
Builds on
31 of 56 references.
References · 31
- 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 266
- 2024Fast Timing-Conditioned Latent Audio Diffusioncited by 178
- 2023Mustango: Toward Controllable Text-to-Music Generationcited by 100
- 2024Tango 2: Aligning Diffusion-based Text-to-Audio Generations through Direct Preference Optimizationcited by 97
- 2024Long-form music generation with latent diffusioncited by 88
- 2023LP-MusicCaps: LLM-Based Pseudo Music Captioningcited by 83
- 2023Music ControlNet: Multiple Time-varying Controls for Music Generationcited by 82
- 2023Music Understanding LLaMA: Advancing Text-to-Music Generation with Question Answering and Captioningcited by 77
- 2024ChatMusician: Understanding and Generating Music Intrinsically with LLMcited by 67
- 2020Learning Interpretable Representation for Controllable Polyphonic Music Generationcited by 40
- 2024Diff-A-Riff: Musical Accompaniment Co-creation via Latent Diffusion Modelscited by 33
- 2024MusicMagus: Zero-Shot Text-to-Music Editing via Diffusion Modelscited by 32
19 more
- 2024Joint Audio and Symbolic Conditioning for Temporally Controlled Text-to-Music Generationcited by 27
- 2023Audio-Text Models Do Not Yet Leverage Natural Languagecited by 26
- 2020AI Song Contest: Human-AI Co-Creation in Songwritingcited by 25
- 2023InstructME: An Instruction Guided Music Edit And Remix Framework with Latent Diffusion Modelscited by 23
- 2020A Differentiable Perceptual Audio Metric Learned from Just Noticeable Differencescited by 22
- 2019Learning Disentangled Representations of Timbre and Pitch for Musical Instrument Sounds Using Gaussian Mixture Variational Autoencoderscited by 21
- 2023Content-based Controls For Music Large Language Modelingcited by 19
- 2024T-CLAP: Temporal-Enhanced Contrastive Language-Audio Pretrainingcited by 18
- 2023JEN-1 Composer: A Unified Framework for High-Fidelity Multi-Track Music Generationcited by 17
- 2024Arrange, Inpaint, and Refine: Steerable Long-term Music Audio Generation and Editing via Content-based Controlscited by 17
- 2023Loop Copilot: Conducting AI Ensembles for Music Generation and Iterative Editingcited by 15
- 2024Instruct-MusicGen: Unlocking Text-to-Music Editing for Music Language Models via Instruction Tuningcited by 15
- 2024Generalized Multi-Source Inference for Text Conditioned Music Diffusion Modelscited by 13
- 2019Perceptual Loss Function for Neural Modelling of Audio Systemscited by 11
- 2024Listen, Chat, and Remix: Text-Guided Soundscape Remixing for Enhanced Auditory Experiencecited by 6
- 2024PAGURI: a user experience study of creative interaction with text-to-music modelscited by 4
- 2023IteraTTA: An interface for exploring both text prompts and audio priors in generating music with text-to-audio modelscited by 3
- 2019Moving to Communicate, Moving to Interact: Patterns of Body Motion in Musical Duo Performancecited by 2
- 2024JEN-1 DreamStyler: Customized Musical Concept Learning via Pivotal Parameters Tuningcited by 1
Built on by
Citing papers per year: 2025: 1.
1 paper.