mlsanityData to 14 Oct 2026

Long-form music generation with latent diffusion

arXiv 2404.10301 ↗First posted 16 Apr 2024cs.SD cs.LG eess.AScited by 89

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

Audio-based generative models for music have seen great strides recently, but so far have not managed to produce full-length music tracks with coherent musical structure from text prompts. We show that by training a generative model on long temporal contexts it is possible to produce long-form music of up to 4m45s. Our model consists of a diffusion-transformer operating on a highly downsampled continuous latent representation (latent rate of 21.5Hz). It obtains state-of-the-art generations according to metrics on audio quality and prompt alignment, and subjective tests reveal that it produces full-length music with coherent structure.

Key facts. Cited by 89 papers in the map (data through 14 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 41 of its 49 references that are in the map. First posted on arXiv 16 Apr 2024.

Builds on

41 of 49 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Bass Accompaniment Generation via Latent Diffusion2 Feb
  2. 2024DITTO: Diffusion Inference-Time T-Optimization for Music Generation22 Jan
  3. 2023Controllable Music Production with Diffusion Models and Guidance Gradients1 Nov
  4. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining10 Aug
  5. 2023Noise2Music: Text-conditioned Music Generation with Diffusion Models8 Feb
  6. 2023Multi-Source Diffusion Models for Simultaneous Music Generation and Separation4 Feb
  7. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models30 Jan
  8. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Models29 Jan
  9. 2021CRASH: Raw Audio Score-based Generative Modeling for Controllable High-resolution Drum Sound Synthesis14 Jun

Other references · 32

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2022Scalable Diffusion Models with Transformerscited by 6,054
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  5. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  7. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
  8. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Modelscited by 938
  9. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  10. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  11. 2023Extracting Training Data from Diffusion Modelscited by 732
  12. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 719
20 more
  1. 2022High Fidelity Neural Audio Compressioncited by 687
  2. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  3. 2020Jukebox: A Generative Model for Musiccited by 542
  4. 2023Simple and Controllable Music Generationcited by 488
  5. 2023MusicLM: Generating Music From Textcited by 448
  6. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 431
  7. 2023Common Diffusion Noise Schedules and Sample Steps are Flawedcited by 310
  8. 2021Efficient Training of Audio Transformers with Patchoutcited by 226
  9. 2016FMA: A Dataset For Music Analysiscited by 207
  10. 2024Fast Timing-Conditioned Latent Audio Diffusioncited by 181
  11. 2022HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detectioncited by 176
  12. 2016SampleRNN: An Unconditional End-to-End Neural Audio Generation Modelcited by 110
  13. 2020Neural Networks Fail to Learn Periodic Functions and How to Fix Itcited by 100
  14. 2023The Song Describer Dataset: a Corpus of Audio Captions for Music-and-Language Evaluationcited by 68
  15. 2023VampNet: Music Generation via Masked Acoustic Token Modelingcited by 57
  16. 2023Efficient Neural Music Generationcited by 50
  17. 2024Masked Audio Generation using a Single Non-Autoregressive Transformercited by 50
  18. 2022Full-band General Audio Synthesis with Score-based Diffusioncited by 32
  19. 2023StemGen: A music generation model that listenscited by 32
  20. 2024MusicRL: Aligning Music Generation to Human Preferencescited by 32

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 27; 2025: 35; 2026: 27.

89 papers.

Sort:

By area:

  1. 2024Stable Audio Opencited by 197
  2. 2024TangoFlux: Super Fast and Faithful Text to Audio Generation with Flow Matching and Clap-Ranked Preference Optimizationcited by 68
  3. 2024A Survey on Diffusion Models for Time Series and Spatio-Temporal Datacited by 67
  4. 2024Scaling Transformers for Low-Bitrate High-Quality Speech Codingcited by 49
  5. 2025DiffRhythm: Blazingly Fast and Embarrassingly Simple End-to-End Full-Length Song Generation with Latent Diffusioncited by 46
  6. 2024Foundation Models for Music: A Surveycited by 37
  7. 2024EzAudio: Enhancing Text-to-Audio Generation with Efficient Diffusion Transformercited by 31
  8. 2025AudioX: A Unified Framework for Anything-to-Audio Generationcited by 27
  9. 2024Both Ears Wide Open: Towards Language-Driven Spatial Audio Generationcited by 25
  10. 2025FreeAudio: Training-Free Timing Planning for Controllable Long-Form Text-to-Audio Generationcited by 21
  11. 2024ETTA: Elucidating the Design Space of Text-to-Audio Modelscited by 17
  12. 2024Presto! Distilling Steps and Layers for Accelerating Music Generationcited by 16
38 more
  1. 2024Instruct-MusicGen: Unlocking Text-to-Music Editing for Music Language Models via Instruction Tuningcited by 15
  2. 2025Fast Text-to-Audio Generation with Adversarial Post-Trainingcited by 14
  3. 2025Audio Super-Resolution with Latent Bridge Modelscited by 14
  4. 2024LoVA: Long-form Video-to-Audio Generationcited by 13
  5. 2024Editing Music with Melody and Text: Using ControlNet for Diffusion Transformercited by 13
  6. 2024ImmerseDiffusion: A Generative Spatial Audio Latent Diffusion Modelcited by 11
  7. 2024Improving Musical Accompaniment Co-creation via Diffusion Transformerscited by 11
  8. 2025InspireMusic: Integrating Super Resolution and Large Language Model for High-Fidelity Long-Form Music Generationcited by 11
  9. 2025ControlAudio: Tackling Text-Guided, Timing-Indicated and Intelligible Audio Generation via Progressive Diffusion Modelingcited by 11
  10. 2025Live Music Modelscited by 10
  11. 2024SoundCTM: Unifying Score-based and Consistency Models for Full-band Text-to-Sound Generationcited by 9
  12. 2024MoLA: Motion Generation and Editing with Latent Diffusion Enhanced by Adversarial Trainingcited by 9
  13. 2025MuseControlLite: Multifunctional Music Generation with Lightweight Conditionerscited by 9
  14. 2024MEDIC: Zero-shot Music Editing with Disentangled Inversion Controlcited by 8
  15. 2025Music Arena: Live Evaluation for Text-to-Musiccited by 8
  16. 2025Language Model Based Text-to-Audio Generation: Anti-Causally Aligned Collaborative Residual Transformerscited by 8
  17. 2025CoDiCodec: Unifying Continuous and Discrete Compressed Representations of Audiocited by 7
  18. 2024Quality-aware Masked Diffusion Transformer for Enhanced Music Generationcited by 6
  19. 2025DiTSE: High-Fidelity Generative Speech Enhancement via Latent Diffusion Transformerscited by 6
  20. 2026Stable Audio 3cited by 6
  21. 2024FolAI: Synchronized Foley Sound Generation with Semantic and Temporal Alignmentcited by 5
  22. 2025Latent Swap Joint Diffusion for 2D Long-Form Latent Generationcited by 5
  23. 2024Multi-Source Music Generation with Latent Diffusioncited by 4
  24. 2025Music2Latent2: Audio Compression with Summary Embeddings and Autoregressive Decodingcited by 4
  25. 2025DRAGON: Distributional Rewards Optimize Diffusion Generative Modelscited by 4
  26. 2025MGE-LDM: Joint Latent Diffusion for Simultaneous Music Generation and Source Extractioncited by 4
  27. 2025VoiceBridge: General Speech Restoration with One-step Latent Bridge Modelscited by 4
  28. 2026Woosh: A Sound Effects Foundation Modelcited by 4
  29. 2025Estimating Musical Surprisal in Audiocited by 3
  30. 2025Generative AI Training and Copyright Lawcited by 3
  31. 2025GD-Retriever: Controllable Generative Text-Music Retrieval with Diffusion Modelscited by 3
  32. 2025Bob's Confetti: Phonetic Memorization Attacks in Music and Video Generationcited by 3
  33. 2026SAME: A Semantically-Aligned Music Autoencodercited by 3
  34. 2025Latent Granular Resynthesis using Neural Audio Codecscited by 2
  35. 2026Guided by the Plan: Enhancing Faithful Autoregressive Text-to-Audio Generation with Guided Decodingcited by 2
  36. 2026Low-Resource Guidance for Controllable Latent Audio Diffusioncited by 2
  37. 2026Few-shot Acoustic Synthesis with Multimodal Flow Matchingcited by 2
  38. 2024JEN-1 DreamStyler: Customized Musical Concept Learning via Pivotal Parameters Tuningcited by 1

50 most cited of 89 All 89 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Multi-Track MusicLDM: Towards Versatile Music Generation with Latent Diffusion Model
  2. 2023Mousai: Text-to-Music Generation with Long-Context Latent Diffusion
  3. 2024Simultaneous Music Separation and Generation Using Multi-Track Latent Diffusion Models
  4. 2026Pitch-class Steering for Diffusion-based Music Generation via Latent-space Probes
  5. 2024Diff-A-Riff: Musical Accompaniment Co-creation via Latent Diffusion Models
  6. 2025ProGress: Structured Music Generation via Graph Diffusion and Hierarchical Music Analysis
  7. 2025Generation of Musical Timbres using a Text-Guided Diffusion Model
  8. 2025PhraseVAE and PhraseLDM: Latent Diffusion for Full-Song Multitrack Symbolic Music Generation