mlsanityData to 7 Oct 2026

Mustango: Toward Controllable Text-to-Music Generation

arXiv 2311.08355 ↗First posted 14 Nov 2023eess.AScited by 100

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

The quality of the text-to-music models has reached new heights due to recent advancements in diffusion models. The controllability of various musical aspects, however, has barely been explored. In this paper, we propose Mustango: a music-domain-knowledge-inspired text-to-music system based on diffusion. Mustango aims to control the generated music, not only with general text captions, but with more rich captions that can include specific instructions related to chords, beats, tempo, and key. At the core of Mustango is MuNet, a Music-Domain-Knowledge-Informed UNet guidance module that steers the generated music to include the music-specific conditions, which we predict from the text prompt, as well as the general text embedding, during the reverse diffusion process. To overcome the limited availability of open datasets of music with text captions, we propose a novel data augmentation method that includes altering the harmonic, rhythmic, and dynamic aspects of music audio and using state-of-the-art Music Information Retrieval methods to extract the music features which will then be appended to the existing descriptions in text format. We release the resulting MusicBench dataset which contains over 52K instances and includes music-theory-based descriptions in the caption text. Through extensive experiments, we show that the quality of the music generated by Mustango is state-of-the-art, and the controllability through music-specific text prompts greatly outperforms other models such as MusicGen and AudioLDM2.

Key facts. Cited by 100 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 22 of its 50 references that are in the map. First posted on arXiv 14 Nov 2023.

Builds on

22 of 50 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining10 Aug
  2. 2023JEN-1: Text-Guided Universal Music Generation with Omnidirectional Diffusion Models9 Aug
  3. 2023Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion Model24 Apr
  4. 2023ERNIE-Music: Text-to-Waveform Music Generation with Diffusion Models9 Feb
  5. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Models29 Jan
  6. 2023Mousai: Text-to-Music Generation with Long-Context Latent Diffusion27 Jan
  7. 2021Diffusion-Based Voice Conversion with Fast Maximum Likelihood Sampling Scheme28 Sep

Other references · 15

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  5. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  6. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  7. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 707
  8. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  9. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 563
  10. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  11. 2016FMA: A Dataset For Music Analysiscited by 205
  12. 2018A Functional Taxonomy of Music Generation Systemscited by 31
3 more
  1. 2022Musika! Fast Infinite Waveform Music Generationcited by 24
  2. 2022A Domain-Knowledge-Inspired Music Embedding Space and a Novel Attention Mechanism for Symbolic Music Modelingcited by 17
  3. 2021BeatNet: CRNN and Particle Filtering for Online Joint Beat Downbeat and Meter Trackingcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 27; 2025: 40; 2026: 33.

100 papers.

Sort:

By area:

  1. 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 281
  2. 2025Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitiescited by 127
  3. 2025MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mixcited by 113
  4. 2024Tango 2: Aligning Diffusion-based Text-to-Audio Generations through Direct Preference Optimizationcited by 97
  5. 2025Audio-Reasoner: Improving Reasoning Capability in Large Audio Language Modelscited by 94
  6. 2024Foundation Models for Music: A Surveycited by 34
  7. 2024MusicMagus: Zero-Shot Text-to-Music Editing via Diffusion Modelscited by 32
  8. 2025SARI: Structured Audio Reasoning via Curriculum-Guided Reinforcement Learningcited by 31
  9. 2024Taming Data and Transformers for Audio Generationcited by 28
  10. 2025InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialoguecited by 27
  11. 2025Uni-MoE-2.0-Omni: Scaling Language-Centric Omnimodal Large Model with Advanced MoE, Training and Datacited by 21
  12. 2024MidiCaps: A large-scale MIDI dataset with text captionscited by 19
38 more
  1. 2024Improving Text-To-Audio Models with Synthetic Captionscited by 19
  2. 2024FLUX that Plays Musiccited by 18
  3. 2024Text2midi: Generating Symbolic Music from Captionscited by 18
  4. 2024ETTA: Elucidating the Design Space of Text-to-Audio Modelscited by 17
  5. 2024MuChin: A Chinese Colloquial Description Benchmark for Evaluating Language Models in the Field of Musiccited by 15
  6. 2024Instruct-MusicGen: Unlocking Text-to-Music Editing for Music Language Models via Instruction Tuningcited by 15
  7. 2025CMI-Bench: A Comprehensive Benchmark for Evaluating Music Instruction Followingcited by 15
  8. 2025Benchmarking Music Generation Models and Metrics via Human Preference Studiescited by 15
  9. 2026Audio Flamingo Next: Next-Generation Open Audio-Language Models for Speech, Sound, and Musiccited by 15
  10. 2025JamendoMaxCaps: A Large Scale Music-caption Dataset with Imputed Metadatacited by 14
  11. 2025SpeechR: A Benchmark for Speech Reasoning in Large Audio-Language Modelscited by 14
  12. 2024Simultaneous Music Separation and Generation Using Multi-Track Latent Diffusion Modelscited by 13
  13. 2024Audio Prompt Adapter: Unleashing Music Editing Abilities for Text-to-Music with Lightweight Finetuningcited by 12
  14. 2024MusiConGen: Rhythm and Chord Control for Transformer-Based Text-to-Music Generationcited by 11
  15. 2025Discovering and Steering Interpretable Concepts in Large Generative Music Modelscited by 11
  16. 2025Benchmarking and Confidence Evaluation of LALMs For Temporal Reasoningcited by 10
  17. 2026Audio-Omni: Extending Multi-modal Understanding to Versatile Audio Generation and Editingcited by 10
  18. 2025MuseControlLite: Multifunctional Music Generation with Lightweight Conditionerscited by 9
  19. 2024Audio-Agent: Leveraging LLMs For Audio Generation, Editing and Compositioncited by 8
  20. 2024Music Consistency Modelscited by 7
  21. 2024Are We There Yet? A Brief Survey of Music Emotion Prediction Datasets, Models and Outstanding Challengescited by 7
  22. 2025SonicMaster: Towards Controllable All-in-One Music Restoration and Masteringcited by 6
  23. 2024Futga: Towards Fine-grained Music Understanding through Temporally-enhanced Generative Augmentationcited by 5
  24. 2024Multi-Track MusicLDM: Towards Versatile Music Generation with Latent Diffusion Modelcited by 5
  25. 2025DanceChat: Large Language Model-Guided Music-to-Dance Generationcited by 5
  26. 2024An Independence-promoting Loss for Music Generation with Language Modelscited by 4
  27. 2025Music for All: Representational Bias and Cross-Cultural Adaptability of Music Generation Modelscited by 4
  28. 2025NOTA: Multimodal Music Notation Understanding for Visual Large Language Modelcited by 4
  29. 2025A Survey on Music Generation from Single-Modal, Cross-Modal, and Multi-Modal Perspectivescited by 4
  30. 2025MusFlow: Multimodal Music Generation via Conditional Flow Matchingcited by 4
  31. 2025Text2midi-InferAlign: Improving Symbolic Music Generation with Inference-Time Alignmentcited by 4
  32. 2025Moonbeam: A MIDI Foundation Model Using Both Absolute and Relative Music Attributescited by 4
  33. 2025Steering Autoregressive Music Generation with Recursive Feature Machinescited by 4
  34. 2024C3LLM: Conditional Multimodal Content Generation Using Large Language Modelscited by 3
  35. 2024Missing Melodies: AI Music Generation and its "Nearly" Complete Omission of the Global Southcited by 3
  36. 2025Diffusion based Text-to-Music Generation with Global and Local Text based Conditioningcited by 3
  37. 2025GD-Retriever: Controllable Generative Text-Music Retrieval with Diffusion Modelscited by 3
  38. 2025Controllable Video-to-Music Generation with Multiple Time-Varying Conditionscited by 3

50 most cited of 100 All 100 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026S2Accompanist: A Semantic-Aware and Structure-Guided Diffusion Model for Music Accompaniment Generation
  2. 2024Generalized Multi-Source Inference for Text Conditioned Music Diffusion Models
  3. 2024Editing Music with Melody and Text: Using ControlNet for Diffusion Transformer
  4. 2025Generation of Musical Timbres using a Text-Guided Diffusion Model
  5. 2023Noise2Music: Text-conditioned Music Generation with Diffusion Models
  6. 2024Melody-Guided Music Generation
  7. 2023MusicLDM: Enhancing Novelty in Text-to-Music Generation Using Beat-Synchronous Mixup Strategies
  8. 2025Diff-TONE: Timestep Optimization for iNstrument Editing in Text-to-Music Diffusion Models