mlsanityData to 14 Oct 2026

Noise2Music: Text-conditioned Music Generation with Diffusion Models

arXiv 2302.03917 ↗First posted 8 Feb 2023cs.SD cs.LG eess.AScited by 173

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

We introduce Noise2Music, where a series of diffusion models is trained to generate high-quality 30-second music clips from text prompts. Two types of diffusion models, a generator model, which generates an intermediate representation conditioned on text, and a cascader model, which generates high-fidelity audio conditioned on the intermediate representation and possibly the text, are trained and utilized in succession to generate high-fidelity music. We explore two options for the intermediate representation, one using a spectrogram and the other using audio with lower fidelity. We find that the generated audio is not only able to faithfully reflect key elements of the text prompt such as genre, tempo, instruments, mood, and era, but goes beyond to ground fine-grained semantics of the prompt. Pretrained large language models play a key role in this story -- they are used to generate paired text for the audio of the training set and to extract embeddings of the text prompts ingested by the diffusion models. Generated examples: https://google-research.github.io/noise2music

Key facts. Cited by 173 papers in the map (data through 14 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 36 of its 44 references that are in the map. First posted on arXiv 8 Feb 2023.

Builds on

36 of 44 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Diffsound: Discrete Diffusion Model for Text-to-sound Generation20 Jul
  2. 2022InferGrad: Improving Diffusion Models for Vocoder by Considering Inference in Training8 Feb
  3. 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech13 May
  4. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesis21 Sep

Other references · 32

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  9. 2022Classifier-Free Diffusion Guidancecited by 5,847
  10. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  11. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  12. 2022Video Diffusion Modelscited by 2,305
20 more
  1. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  2. 2021Palette: Image-to-Image Diffusion Modelscited by 1,300
  3. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  4. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  5. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,163
  6. 2016CNN Architectures for Large-Scale Audio Classificationcited by 704
  7. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 622
  8. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  9. 2020Jukebox: A Generative Model for Musiccited by 542
  10. 2023MusicLM: Generating Music From Textcited by 448
  11. 2016YouTube-8M: A Large-Scale Video Classification Benchmarkcited by 324
  12. 2022AudioGen: Textually Guided Audio Generationcited by 295
  13. 2018Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesiscited by 259
  14. 2018Fr'echet Audio Distance: A Metric for Evaluating Music Enhancement Algorithmscited by 225
  15. 2021GSPMD: General and Scalable Parallelization for ML Computation Graphscited by 111
  16. 2022MuLan: A Joint Embedding of Music Audio and Natural Languagecited by 87
  17. 2020Towards Learning a Universal Non-Semantic Representation of Speechcited by 84
  18. 2022Contrastive Audio-Language Learning for Musiccited by 33
  19. 2020From Artificial Neural Networks to Deep Learning for Music Generation -- History, Concepts and Trendscited by 14
  20. 2021ItoTTS and ItoWave: Linear Stochastic Differential Equation Is All You Need For Audio Generationcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 53; 2024: 70; 2025: 37; 2026: 13.

173 papers.

Sort:

By area:

  1. 2023Simple and Controllable Music Generationcited by 488
  2. 2023Shap-E: Generating Conditional 3D Implicit Functionscited by 385
  3. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretrainingcited by 344
  4. 2023Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystackcited by 279
  5. 2023Efficient Diffusion Training via Min-SNR Weighting Strategycited by 246
  6. 2023Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Modelscited by 245
  7. 2024Fast Timing-Conditioned Latent Audio Diffusioncited by 181
  8. 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 157
  9. 2023CADS: Unleashing the Diversity of Diffusion Models through Condition-Annealed Samplingcited by 129
  10. 2023Audiobox: Unified Audio Generation with Natural Language Promptscited by 120
  11. 2024Eliminating Oversaturation and Artifacts of High Guidance Scales in Diffusion Modelscited by 114
  12. 2023In-Context Learning Unlocked for Diffusion Modelscited by 104
38 more
  1. 2023DreamHuman: Animatable 3D Avatars from Textcited by 102
  2. 2023MusicLDM: Enhancing Novelty in Text-to-Music Generation Using Beat-Synchronous Mixup Strategiescited by 94
  3. 2024Long-form music generation with latent diffusioncited by 89
  4. 2023Language Conditioned Traffic Generationcited by 87
  5. 2023AUDIT: Audio Editing by Following Instructions with Latent Diffusion Modelscited by 83
  6. 2024DITTO: Diffusion Inference-Time T-Optimization for Music Generationcited by 73
  7. 2023The Song Describer Dataset: a Corpus of Audio Captions for Music-and-Language Evaluationcited by 68
  8. 2023One-Step Diffusion Distillation via Deep Equilibrium Modelscited by 61
  9. 2023On the Design Fundamentals of Diffusion Models: A Surveycited by 60
  10. 2024Symbolic Music Generation with Non-Differentiable Rule Guided Diffusioncited by 54
  11. 2024No Training, No Problem: Rethinking Classifier-Free Guidance for Diffusion Modelscited by 53
  12. 2023MuseCoco: Generating Symbolic Music from Textcited by 52
  13. 2023Efficient Neural Music Generationcited by 50
  14. 2023ControlMat: A Controlled Generative Approach to Material Capturecited by 50
  15. 2024Masked Audio Generation using a Single Non-Autoregressive Transformercited by 50
  16. 2023Diff2Lip: Audio Conditioned Diffusion Models for Lip-Synchronizationcited by 49
  17. 2023Instruct 3D-to-3D: Text Instruction Guided 3D-to-3D conversioncited by 48
  18. 2023JEN-1: Text-Guided Universal Music Generation with Omnidirectional Diffusion Modelscited by 41
  19. 2024VidMuse: A Simple Video-to-Music Generation Framework with Long-Short-Term Modelingcited by 40
  20. 2024Text-to-Image Rectified Flow as Plug-and-Play Priorscited by 38
  21. 2024Foundation Models for Music: A Surveycited by 37
  22. 2023Anticipatory Music Transformercited by 36
  23. 2024Tell What You Hear From What You See -- Video to Audio Generation Through Textcited by 34
  24. 2024Diff-A-Riff: Musical Accompaniment Co-creation via Latent Diffusion Modelscited by 33
  25. 2023StemGen: A music generation model that listenscited by 32
  26. 2024MusicRL: Aligning Music Generation to Human Preferencescited by 32
  27. 2024On the Challenges and Opportunities in Generative AIcited by 31
  28. 2025SongGen: A Single Stage Auto-regressive Transformer for Text-to-Song Generationcited by 29
  29. 2023Diffusion-based Image Translation with Label Guidance for Domain Adaptive Semantic Segmentationcited by 27
  30. 2023Discovering Failure Modes of Text-guided Diffusion Models via Adversarial Searchcited by 25
  31. 2023InstructME: An Instruction Guided Music Edit And Remix Framework with Latent Diffusion Modelscited by 23
  32. 2023From Discrete Tokens to High-Fidelity Audio Using Multi-Band Diffusioncited by 22
  33. 2023ConsistencyTTA: Accelerating Diffusion-Based Text-to-Audio Generation with Consistency Distillationcited by 22
  34. 2023HeadArtist: Text-conditioned 3D Head Generation with Self Score Distillationcited by 22
  35. 2024LiteVAE: Lightweight and Efficient Variational Autoencoders for Latent Diffusion Modelscited by 22
  36. 2024DKDM: Data-Free Knowledge Distillation for Diffusion Models with Any Architecturecited by 22
  37. 2023Blind Audio Bandwidth Extension: A Diffusion-Based Zero-Shot Approachcited by 21
  38. 2023Context Diffusion: In-Context Aware Image Generationcited by 21

50 most cited of 173 All 173 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Generalized Multi-Source Inference for Text Conditioned Music Diffusion Models
  2. 2025Generation of Musical Timbres using a Text-Guided Diffusion Model
  3. 2026Instrumental Text-to-Music Generation with Auxiliary Conditioning Branches ↗
  4. 2025Diffusion based Text-to-Music Generation with Global and Local Text based Conditioning
  5. 2023Mousai: Text-to-Music Generation with Long-Context Latent Diffusion
  6. 2021Symbolic Music Generation with Diffusion Models
  7. 2023ArchiSound: Audio Generation with Diffusion
  8. 2025DiffRhythm: Blazingly Fast and Embarrassingly Simple End-to-End Full-Length Song Generation with Latent Diffusion