mlsanityData to 7 Oct 2026

MusicLM: Generating Music From Text

arXiv 2301.11325 ↗First posted 26 Jan 2023cs.SD cs.LG eess.AScited by 442

Not on the map. Why?

Cited by 72 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce MusicLM, a model generating high-fidelity music from text descriptions such as "a calming violin melody backed by a distorted guitar riff". MusicLM casts the process of conditional music generation as a hierarchical sequence-to-sequence modeling task, and it generates music at 24 kHz that remains consistent over several minutes. Our experiments show that MusicLM outperforms previous systems both in audio quality and adherence to the text description. Moreover, we demonstrate that MusicLM can be conditioned on both text and a melody in that it can transform whistled and hummed melodies according to the style described in a text caption. To support future research, we publicly release MusicCaps, a dataset composed of 5.5k music-text pairs, with rich text descriptions provided by human experts.

Key facts. Cited by 442 papers in the map (data through 7 Oct 2026). Builds on 41 of its 52 references that are in the map. First posted on arXiv 26 Jan 2023.

Builds on

41 of 52 references.

References · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  7. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  8. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  11. 2017Neural Discrete Representation Learningcited by 4,650
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,276
29 more
  1. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  2. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  3. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
  4. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,277
  5. 2020Extracting Training Data from Large Language Modelscited by 2,144
  6. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  7. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  8. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  9. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  10. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  11. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  12. 2016CNN Architectures for Large-Scale Audio Classificationcited by 701
  13. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  14. 2020Jukebox: A Generative Model for Musiccited by 535
  15. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  16. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  17. 2016YouTube-8M: A Large-Scale Video Classification Benchmarkcited by 324
  18. 2022AudioGen: Textually Guided Audio Generationcited by 292
  19. 2022Diffsound: Discrete Diffusion Model for Text-to-sound Generationcited by 256
  20. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  21. 2018Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Datasetcited by 215
  22. 2021Wav2CLIP: Learning Robust Audio Representations From CLIPcited by 210
  23. 2018Music Transformercited by 207
  24. 2016FMA: A Dataset For Music Analysiscited by 205
  25. 2020DDSP: Differentiable Digital Signal Processingcited by 177
  26. 2020Towards Learning a Universal Non-Semantic Representation of Speechcited by 84
  27. 2021LEAF: A Learnable Frontend for Audio Classificationcited by 69
  28. 2022General-purpose, long-context autoregressive modeling with Perceiver ARcited by 60
  29. 2022Multi-instrument Music Synthesis with Spectrogram Diffusioncited by 42

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 98; 2024: 138; 2025: 120; 2026: 86.

442 papers.

Sort:

By area:

  1. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  2. 2024Qwen2-Audio Technical Reportcited by 543
  3. 2023Simple and Controllable Music Generationcited by 480
  4. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 479
  5. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 423
  6. 2023Shap-E: Generating Conditional 3D Implicit Functionscited by 379
  7. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretrainingcited by 343
  8. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  9. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  10. 2023Towards Generalist Biomedical AIcited by 286
  11. 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 281
  12. 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 266
38 more
  1. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 264
  2. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Taskscited by 243
  3. 2023Self-Consuming Generative Models Go MADcited by 239
  4. 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Trickedcited by 226
  5. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modelingcited by 213
  6. 2023On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspectivecited by 208
  7. 2023Multimodal Large Language Models: A Surveycited by 193
  8. 2024Stable Audio Opencited by 192
  9. 2024Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilitiescited by 183
  10. 2023Noise2Music: Text-conditioned Music Generation with Diffusion Modelscited by 173
  11. 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 154
  12. 2024GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilitiescited by 151
  13. 2023Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion Modelcited by 145
  14. 2024Lumina-Next: Making Lumina-T2X Stronger and Faster with Next-DiTcited by 130
  15. 2024WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modelingcited by 124
  16. 2023Audiobox: Unified Audio Generation with Natural Language Promptscited by 120
  17. 2023Generative AIcited by 119
  18. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 116
  19. 2024Large Language Models for Time Series: A Surveycited by 113
  20. 2023DreamHuman: Animatable 3D Avatars from Textcited by 102
  21. 2023Textually Pretrained Speech Language Modelscited by 99
  22. 2023STEVE-1: A Generative Model for Text-to-Behavior in Minecraftcited by 95
  23. 2023MusicLDM: Enhancing Novelty in Text-to-Music Generation Using Beat-Synchronous Mixup Strategiescited by 94
  24. 2023Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLMcited by 92
  25. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  26. 2023HumanNorm: Learning Normal Diffusion Model for High-quality and Realistic 3D Human Generationcited by 88
  27. 2024Long-form music generation with latent diffusioncited by 88
  28. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 88
  29. 2023AUDIT: Audio Editing by Following Instructions with Latent Diffusion Modelscited by 81
  30. 2024Red-Teaming for Generative AI: Silver Bullet or Security Theater?cited by 81
  31. 2025Ola: Pushing the Frontiers of Omni-Modal Language Modelcited by 79
  32. 2023Music Understanding LLaMA: Advancing Text-to-Music Generation with Question Answering and Captioningcited by 77
  33. 2024Detecting Multimedia Generated by Large AI Models: A Surveycited by 73
  34. 2023Level Generation Through Large Language Modelscited by 68
  35. 2025MuQ: Self-Supervised Music Representation Learning with Mel Residual Vector Quantizationcited by 66
  36. 2023SpeechX: Neural Codec Language Model as a Versatile Speech Transformercited by 64
  37. 2023Multi-Source Diffusion Models for Simultaneous Music Generation and Separationcited by 63
  38. 2023Large AI Models in Health Informatics: Applications, Challenges, and the Futurecited by 61

50 most cited of 442 All 442 (JSON)