mlsanityData to 7 Oct 2026

Controllable Music Production with Diffusion Models and Guidance Gradients

arXiv 2311.00613 ↗First posted 1 Nov 2023cs.SD cs.LG eess.AScited by 29

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

We demonstrate how conditional generation from diffusion models can be used to tackle a variety of realistic tasks in the production of music in 44.1kHz stereo audio with sampling-time guidance. The scenarios we consider include continuation, inpainting and regeneration of musical audio, the creation of smooth transitions between two different music tracks, and the transfer of desired stylistic characteristics to existing audio clips. We achieve this by applying guidance at sampling time in a simple framework that supports both reconstruction and classification losses, or any combination of the two. This approach ensures that generated audio can match its surrounding context, or conform to a class distribution or latent representation specified relative to any suitable pre-trained classifier or embedding model. Audio samples are available at https://machinelearning.apple.com/research/controllable-music

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 30 of its 37 references that are in the map. First posted on arXiv 1 Nov 2023.

Builds on

30 of 37 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Solving Audio Inverse Problems with a Diffusion Model27 Oct
  2. 2021CRASH: Raw Audio Score-based Generative Modeling for Controllable High-resolution Drum Sound Synthesis14 Jun

Other references · 28

  1. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2020Denoising Diffusion Implicit Modelscited by 9,655
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  6. 2022Scalable Diffusion Models with Transformerscited by 5,931
  7. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  8. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  9. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  10. 2022A ConvNet for the 2020scited by 3,545
  11. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  12. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
16 more
  1. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,172
  2. 2018Self-Attention with Relative Position Representationscited by 1,112
  3. 2022Denoising Diffusion Restoration Modelscited by 951
  4. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 925
  5. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 924
  6. 2016CNN Architectures for Large-Scale Audio Classificationcited by 701
  7. 2020Jukebox: A Generative Model for Musiccited by 535
  8. 2021Diffusion Autoencoders: Toward a Meaningful and Decodable Representationcited by 447
  9. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  10. 2019Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogramcited by 337
  11. 2021Score-based diffusion models for accelerated MRIcited by 308
  12. 2021Efficient Training of Audio Transformers with Patchoutcited by 222
  13. 2018Fr'echet Audio Distance: A Metric for Evaluating Music Enhancement Algorithmscited by 221
  14. 2016FMA: A Dataset For Music Analysiscited by 205
  15. 2021Taming Visually Guided Sound Generationcited by 136
  16. 2022General-purpose, long-context autoregressive modeling with Perceiver ARcited by 60

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 16; 2025: 8; 2026: 5.

29 papers.

Sort:

By area:

  1. 2024Stable Audio Opencited by 192
  2. 2024Fast Timing-Conditioned Latent Audio Diffusioncited by 178
  3. 2024TFG: Unified Training-Free Guidance for Diffusion Modelscited by 103
  4. 2024Long-form music generation with latent diffusioncited by 88
  5. 2024DITTO: Diffusion Inference-Time T-Optimization for Music Generationcited by 73
  6. 2024Diff-A-Riff: Musical Accompaniment Co-creation via Latent Diffusion Modelscited by 33
  7. 2024DITTO-2: Distilled Diffusion Inference-Time T-Optimization for Music Generationcited by 26
  8. 2024Both Ears Wide Open: Towards Language-Driven Spatial Audio Generationcited by 25
  9. 2024Boosting Latent Diffusion with Perceptual Objectivescited by 19
  10. 2024On Improved Conditioning Mechanisms and Pre-training Strategies for Diffusion Modelscited by 12
  11. 2024ImmerseDiffusion: A Generative Spatial Audio Latent Diffusion Modelcited by 11
  12. 2024Improving Musical Accompaniment Co-creation via Diffusion Transformerscited by 11
17 more
  1. 2025Entropy Rectifying Guidance for Diffusion and Flow Modelscited by 11
  2. 2024SoundCTM: Unifying Score-based and Consistency Models for Full-band Text-to-Sound Generationcited by 9
  3. 2025MuseControlLite: Multifunctional Music Generation with Lightweight Conditionerscited by 9
  4. 2025Live Music Modelscited by 9
  5. 2024Tri-Ergon: Fine-grained Video-to-Audio Generation with Multi-modal Conditions and LUFS Controlcited by 7
  6. 2026Stable Audio 3cited by 5
  7. 2024Greedy Growing Enables High-Resolution Pixel-Based Diffusion Modelscited by 3
  8. 2025LiLAC: A Lightweight Latent ControlNet for Musical Audio Generationcited by 3
  9. 2025Post-Training Quantization for Audio Diffusion Transformerscited by 2
  10. 2025Flowception: Temporally Expansive Flow Matching for Video Generationcited by 2
  11. 2026Few-shot Acoustic Synthesis with Multimodal Flow Matchingcited by 2
  12. 2025Audio Generation Through Score-Based Generative Modeling: Design Principles and Implementationcited by 1
  13. 2026Low-Resource Guidance for Controllable Latent Audio Diffusioncited by 1
  14. 2024Interpreting Graphic Notation with MusicLDM: An AI Improvisation of Cornelius Cardew's Treatisecited by 0
  15. 2025Similarity-Guided Diffusion for Long-Gap Music Inpaintingcited by 0
  16. 2026Latent Fourier Transformcited by 0
  17. 2026Break-the-Beat! Controllable MIDI-to-Drum Audio Synthesiscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Bass Accompaniment Generation via Latent Diffusion
  2. 2026Diff2Mix: Controllable Music Mixing via Diffusion Models and Differentiable Audio Effects
  3. 2024Combining audio control and style transfer using latent diffusion
  4. 2026Pitch-class Steering for Diffusion-based Music Generation via Latent-space Probes
  5. 2026Live Music Diffusion Models: Efficient Fine-Tuning and Post-Training of Interactive Diffusion Music Generators
  6. 2025Conditional Diffusion as Latent Constraints for Controllable Symbolic Music Generation
  7. 2023EDMSound: Spectrogram Based Diffusion Models for Efficient and High-Quality Audio Synthesis
  8. 2023Multi-Source Diffusion Models for Simultaneous Music Generation and Separation