mlsanityData to 14 Oct 2026

Simple and Controllable Music Generation

arXiv 2306.05284 ↗First posted 8 Jun 2023cs.SD cs.AI cs.LGcited by 488

Not on the map. Why?

Cited by 87 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We tackle the task of conditional music generation. We introduce MusicGen, a single Language Model (LM) that operates over several streams of compressed discrete music representation, i.e., tokens. Unlike prior work, MusicGen is comprised of a single-stage transformer LM together with efficient token interleaving patterns, which eliminates the need for cascading several models, e.g., hierarchically or upsampling. Following this approach, we demonstrate how MusicGen can generate high-quality samples, both mono and stereo, while being conditioned on textual description or melodic features, allowing better controls over the generated output. We conduct extensive empirical evaluation, considering both automatic and human studies, showing the proposed approach is superior to the evaluated baselines on a standard text-to-music benchmark. Through ablation studies, we shed light over the importance of each of the components comprising MusicGen. Music samples, code, and models are available at https://github.com/facebookresearch/audiocraft

Key facts. Cited by 488 papers in the map (data through 14 Oct 2026). Builds on 37 of its 49 references that are in the map. First posted on arXiv 8 Jun 2023.

Builds on

37 of 49 references.

References · 37

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2016Layer Normalizationcited by 5,669
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  8. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  9. 2018Hierarchical Neural Story Generationcited by 1,193
  10. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  11. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  12. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 719
25 more
  1. 2022High Fidelity Neural Audio Compressioncited by 687
  2. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Modelscited by 599
  3. 2020Jukebox: A Generative Model for Musiccited by 542
  4. 2023MusicLM: Generating Music From Textcited by 448
  5. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  6. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Modelscited by 324
  7. 2022AudioGen: Textually Guided Audio Generationcited by 295
  8. 2022Diffsound: Discrete Diffusion Model for Text-to-sound Generationcited by 257
  9. 2023A Cookbook of Self-Supervised Learningcited by 250
  10. 2021Efficient Training of Audio Transformers with Patchoutcited by 226
  11. 2018Fr'echet Audio Distance: A Metric for Evaluating Music Enhancement Algorithmscited by 225
  12. 2021A Survey on Neural Speech Synthesiscited by 184
  13. 2023Noise2Music: Text-conditioned Music Generation with Diffusion Modelscited by 173
  14. 2017MuseGAN: Multi-track Sequential Generative Adversarial Networks for Symbolic Music Generation and Accompanimentcited by 151
  15. 2021Text-Free Prosody-Aware Generative Spoken Language Modelingcited by 119
  16. 2023Learning-Rate-Free Learning by D-Adaptationcited by 113
  17. 2019Music Source Separation in the Waveform Domaincited by 99
  18. 2022I Hear Your True Colors: Image Guided Audio Generationcited by 92
  19. 2022MuLan: A Joint Embedding of Music Audio and Natural Languagecited by 87
  20. 2020Foley Music: Learning to Generate Music from Videoscited by 83
  21. 2023Mousai: Text-to-Music Generation with Long-Context Latent Diffusioncited by 78
  22. 2020A Comprehensive Survey on Deep Music Generation: Multi-level Representations, Algorithms, Evaluations, and Future Directionscited by 56
  23. 2023SingSong: Generating musical accompaniments from singingcited by 55
  24. 2023Msanii: High Fidelity Music Synthesis on a Shoestring Budgetcited by 6
  25. 2022Unsupervised Symbolic Music Segmentation using Ensemble Temporal Prediction Errorscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 32; 2024: 164; 2025: 164; 2026: 128.

488 papers.

Sort:

By area:

  1. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 568
  2. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretrainingcited by 344
  3. 2024Discrete Flow Matchingcited by 317
  4. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Taskscited by 246
  5. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modelingcited by 217
  6. 2024Mini-Omni: Language Models Can Hear, Talk While Thinking in Streamingcited by 210
  7. 2024Stable Audio Opencited by 197
  8. 2024Fast Timing-Conditioned Latent Audio Diffusioncited by 181
  9. 2024MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformercited by 161
  10. 2024SAFREE: Training-Free and Adaptive Guard for Safe Text-to-Image And Video Generationcited by 133
  11. 2024WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modelingcited by 129
  12. 2024D-Flow: Differentiating through Flows for Controlled Generationcited by 122
38 more
  1. 2023Audiobox: Unified Audio Generation with Natural Language Promptscited by 120
  2. 2024VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wildcited by 120
  3. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 118
  4. 2025Step-Audio 2 Technical Reportcited by 105
  5. 2023HumanTOMATO: Text-aligned Whole-body Motion Generationcited by 104
  6. 2023MusicLDM: Enhancing Novelty in Text-to-Music Generation Using Beat-Synchronous Mixup Strategiescited by 94
  7. 2024Long-form music generation with latent diffusioncited by 89
  8. 2025STream3R: Scalable Sequential 3D Reconstruction with Causal Transformercited by 82
  9. 2023LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPTcited by 78
  10. 2024Proactive Detection of Voice Cloning with Localized Watermarkingcited by 78
  11. 2023Music Understanding LLaMA: Advancing Text-to-Music Generation with Question Answering and Captioningcited by 77
  12. 2024DITTO: Diffusion Inference-Time T-Optimization for Music Generationcited by 73
  13. 2023Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptationcited by 69
  14. 2024Natural language guidance of high-fidelity text-to-speech with synthetic annotationscited by 67
  15. 2025MiMo-Audio: Audio Language Models are Few-Shot Learnerscited by 65
  16. 2023VampNet: Music Generation via Masked Acoustic Token Modelingcited by 57
  17. 2024Temporally Aligned Audio for Video with Autoregressioncited by 52
  18. 2023WavMark: Watermarking for Audio Generationcited by 50
  19. 2024Masked Audio Generation using a Single Non-Autoregressive Transformercited by 50
  20. 2024Scaling Transformers for Low-Bitrate High-Quality Speech Codingcited by 49
  21. 2024SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General Soundcited by 45
  22. 2024SNAC: Multi-Scale Neural Audio Codeccited by 43
  23. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 43
  24. 2024ComposerX: Multi-Agent Symbolic Music Composition with LLMscited by 42
  25. 2023JEN-1: Text-Guided Universal Music Generation with Omnidirectional Diffusion Modelscited by 41
  26. 2024VidMuse: A Simple Video-to-Music Generation Framework with Long-Short-Term Modelingcited by 40
  27. 2023AudioSR: Versatile Audio Super-resolution at Scalecited by 39
  28. 2024VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignmentcited by 38
  29. 2024Foundation Models for Music: A Surveycited by 37
  30. 2025MotionGPT3: Human Motion as a Second Modalitycited by 37
  31. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  32. 2024Autoregressive Diffusion Transformer for Text-to-Speech Synthesiscited by 34
  33. 2024Tell What You Hear From What You See -- Video to Audio Generation Through Textcited by 34
  34. 2025ThinkSound: Chain-of-Thought Reasoning in Multimodal Large Language Models for Audio Generation and Editingcited by 34
  35. 2024MuPT: A Generative Symbolic Music Pretrained Transformercited by 33
  36. 2024Diff-A-Riff: Musical Accompaniment Co-creation via Latent Diffusion Modelscited by 33
  37. 2023StemGen: A music generation model that listenscited by 32
  38. 2024MusicMagus: Zero-Shot Text-to-Music Editing via Diffusion Modelscited by 32

50 most cited of 488 All 488 (JSON)