mlsanityData to 14 Oct 2026

DITTO: Diffusion Inference-Time T-Optimization for Music Generation

arXiv 2401.12179 ↗First posted 22 Jan 2024cs.SD cs.AI cs.LGcited by 73

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

We propose Diffusion Inference-Time T-Optimization (DITTO), a general-purpose frame-work for controlling pre-trained text-to-music diffusion models at inference-time via optimizing initial noise latents. Our method can be used to optimize through any differentiable feature matching loss to achieve a target (stylized) output and leverages gradient checkpointing for memory efficiency. We demonstrate a surprisingly wide-range of applications for music generation including inpainting, outpainting, and looping as well as intensity, melody, and musical structure control - all without ever fine-tuning the underlying model. When we compare our approach against related training, guidance, and optimization-based methods, we find DITTO achieves state-of-the-art performance on nearly all tasks, including outperforming comparable approaches on controllability, audio quality, and computational efficiency, thus opening the door for high-quality, flexible, training-free control of diffusion models. Sound examples can be found at https://DITTO-Music.github.io/web/.

Key facts. Cited by 73 papers in the map (data through 14 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 67 of its 85 references that are in the map. First posted on arXiv 22 Jan 2024.

Builds on

67 of 85 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023Controllable Music Production with Diffusion Models and Guidance Gradients1 Nov
  2. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining10 Aug
  3. 2023MusicLDM: Enhancing Novelty in Text-to-Music Generation Using Beat-Synchronous Mixup Strategies3 Aug
  4. 2023Noise2Music: Text-conditioned Music Generation with Diffusion Models8 Feb
  5. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models30 Jan
  6. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Models29 Jan
  7. 2023Mousai: Text-to-Music Generation with Long-Context Latent Diffusion27 Jan
  8. 2022Multi-instrument Music Synthesis with Spectrogram Diffusion11 Jun

Other references · 59

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2020Denoising Diffusion Implicit Modelscited by 9,781
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  8. 2022Classifier-Free Diffusion Guidancecited by 5,847
  9. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  10. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  11. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
  12. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
38 more
  1. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  2. 2022Video Diffusion Modelscited by 2,305
  3. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  4. 2016Density estimation using Real NVPcited by 2,042
  5. 2022Diffusion Posterior Sampling for General Noisy Inverse Problemscited by 1,416
  6. 2021Palette: Image-to-Image Diffusion Modelscited by 1,300
  7. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,179
  8. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,178
  9. 2023Training Diffusion Models with Reinforcement Learningcited by 982
  10. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Modelscited by 938
  11. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  12. 2023Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inferencecited by 846
  13. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  14. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 719
  15. 2016CNN Architectures for Large-Scale Audio Classificationcited by 704
  16. 2022High Fidelity Neural Audio Compressioncited by 687
  17. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  18. 2020Jukebox: A Generative Model for Musiccited by 542
  19. 2023MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generationcited by 536
  20. 2023Simple and Controllable Music Generationcited by 488
  21. 2023Consistency Trajectory Models: Learning Probability Flow ODE Trajectory of Diffusioncited by 459
  22. 2021Diffusion Autoencoders: Toward a Meaningful and Decodable Representationcited by 449
  23. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewardscited by 448
  24. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 431
  25. 2021GAN Inversion: A Surveycited by 367
  26. 2023Uni-ControlNet: All-in-One Control to Text-to-Image Diffusion Modelscited by 359
  27. 2023Photorealistic Video Generation with Diffusion Modelscited by 306
  28. 2022Novel View Synthesis with Diffusion Modelscited by 300
  29. 2023FreeDoM: Training-Free Energy-Guided Conditional Diffusion Modelcited by 294
  30. 2023Aligning Text-to-Image Diffusion Models with Reward Backpropagationcited by 288
  31. 2022EDICT: Exact Diffusion Inversion via Coupled Transformationscited by 258
  32. 2022BigVGAN: A Universal Neural Vocoder with Large-Scale Trainingcited by 251
  33. 2018Fr'echet Audio Distance: A Metric for Evaluating Music Enhancement Algorithmscited by 225
  34. 2016FMA: A Dataset For Music Analysiscited by 207
  35. 2023FreeU: Free Lunch in Diffusion U-Netcited by 196
  36. 2023On the Importance of Noise Scheduling for Diffusion Modelscited by 194
  37. 2023SoundStorm: Efficient Parallel Audio Generationcited by 114
  38. 2023End-to-End Diffusion Latent Optimization Improves Classifier Guidancecited by 108

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 22; 2025: 31; 2026: 20.

73 papers.

Sort:

By area:

  1. 2025Inference-Time Scaling for Diffusion Models beyond Scaling Denoising Stepscited by 234
  2. 2024ReNO: Enhancing One-step Text-to-Image Models through Reward-based Noise Optimizationcited by 119
  3. 2024Long-form music generation with latent diffusioncited by 89
  4. 2024Inference-Time Alignment of Diffusion Models with Direct Noise Optimizationcited by 55
  5. 2025Noise Hypernetworks: Amortizing Test-Time Compute in Diffusion Modelscited by 35
  6. 2024Diff-A-Riff: Musical Accompaniment Co-creation via Latent Diffusion Modelscited by 33
  7. 2024MusicMagus: Zero-Shot Text-to-Music Editing via Diffusion Modelscited by 32
  8. 2024DITTO-2: Distilled Diffusion Inference-Time T-Optimization for Music Generationcited by 26
  9. 2025Aligning Text-to-Music Evaluation with Human Preferencescited by 21
  10. 2024Presto! Distilling Steps and Layers for Accelerating Music Generationcited by 16
  11. 2024PDMX: A Large-Scale Public Domain MusicXML Dataset for Symbolic Music Processingcited by 15
  12. 2024High Fidelity Text-Guided Music Editing via Single-Stage Flow Matchingcited by 13
38 more
  1. 2024Editing Music with Melody and Text: Using ControlNet for Diffusion Transformercited by 13
  2. 2025Source-Guided Flow Matchingcited by 13
  3. 2025CoDA: Coordinated Diffusion Noise Optimization for Whole-Body Manipulation of Articulated Objectscited by 12
  4. 2024Improving Musical Accompaniment Co-creation via Diffusion Transformerscited by 11
  5. 2025It's Never Too Late: Noise Optimization for Collapse Recovery in Trained Diffusion Modelscited by 11
  6. 2025Live Music Modelscited by 10
  7. 2024SoundCTM: Unifying Score-based and Consistency Models for Full-band Text-to-Sound Generationcited by 9
  8. 2025MuseControlLite: Multifunctional Music Generation with Lightweight Conditionerscited by 9
  9. 2024MEDIC: Zero-shot Music Editing with Disentangled Inversion Controlcited by 8
  10. 2025DiTSE: High-Fidelity Generative Speech Enhancement via Latent Diffusion Transformerscited by 6
  11. 2025Steering Autoregressive Music Generation with Recursive Feature Machinescited by 6
  12. 2026Stable Audio 3cited by 6
  13. 2024Futga: Towards Fine-grained Music Understanding through Temporally-enhanced Generative Augmentationcited by 5
  14. 2024Multi-Track MusicLDM: Towards Versatile Music Generation with Latent Diffusion Modelcited by 5
  15. 2025MOSAIC: Generating Consistent, Privacy-Preserving Scenes from Multiple Depth Views in Multi-Room Environmentscited by 5
  16. 2025A Survey on Music Generation from Single-Modal, Cross-Modal, and Multi-Modal Perspectivescited by 4
  17. 2025DRAGON: Distributional Rewards Optimize Diffusion Generative Modelscited by 4
  18. 2025ZeroSep: Separate Anything in Audio with Zero Trainingcited by 4
  19. 2025TITAN-Guide: Taming Inference-Time AligNment for Guided Text-to-Video Diffusion Modelscited by 4
  20. 2026Gencho: Room Impulse Response Generation from Reverberant Speech and Text via Diffusion Transformerscited by 4
  21. 2026TuneJury: An Open Metric for Improving Music Generation Preference Alignmentcited by 4
  22. 2025Repurposing Marigold for Zero-Shot Metric Depth Estimation via Defocus Blur Cuescited by 3
  23. 2025LiLAC: A Lightweight Latent ControlNet for Musical Audio Generationcited by 3
  24. 2025GD-Retriever: Controllable Generative Text-Music Retrieval with Diffusion Modelscited by 3
  25. 2025Bob's Confetti: Phonetic Memorization Attacks in Music and Video Generationcited by 3
  26. 2026Diffusion Timbre Transfer Via Mutual Information Guided Inpaintingcited by 3
  27. 2024AnyLogo: Symbiotic Subject-Driven Diffusion System with Gemini Statuscited by 2
  28. 2024Harmonizing Pixels and Melodies: Maestro-Guided Film Score Generation and Composition Style Transfercited by 2
  29. 2025Greed is Good: A Unifying Perspective on Guided Generationcited by 2
  30. 2026Low-Resource Guidance for Controllable Latent Audio Diffusioncited by 2
  31. 2026V2M-Zero: Zero-Pair Time-Aligned Video-to-Music Generationcited by 2
  32. 2024Efficient Fine-Grained Guidance for Diffusion Model Based Symbolic Music Generationcited by 1
  33. 2025Assessing the Effectiveness of Membership Inference on Generative Musiccited by 1
  34. 2026Stemphonic: All-at-once Flexible Multi-stem Music Generationcited by 1
  35. 2026LENS: Low-Frequency Eigen Noise Shaping for Efficient Diffusion Samplingcited by 1
  36. 2026Live Music Diffusion Models: Efficient Fine-Tuning and Post-Training of Interactive Diffusion Music Generatorscited by 1
  37. 2026StressDream: Steering Video World Models for Robust Policy Evaluation and Improvementcited by 1
  38. 2026NoisEasier: Test-Time Noise Optimization for Text-to-Video Generationcited by 1

50 most cited of 73 All 73 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Diff-TONE: Timestep Optimization for iNstrument Editing in Text-to-Music Diffusion Models
  2. 2025DiffRhythm: Blazingly Fast and Embarrassingly Simple End-to-End Full-Length Song Generation with Latent Diffusion
  3. 2025Conditional Diffusion as Latent Constraints for Controllable Symbolic Music Generation
  4. 2023ERNIE-Music: Text-to-Waveform Music Generation with Diffusion Models
  5. 2024WaveTransfer: A Flexible End-to-end Multi-instrument Timbre Transfer with Diffusion
  6. 2024Generalized Multi-Source Inference for Text Conditioned Music Diffusion Models
  7. 2025Auto-Regressive vs Flow-Matching: a Comparative Study of Modeling Paradigms for Text-to-Music Generation
  8. 2024Simultaneous Music Separation and Generation Using Multi-Track Latent Diffusion Models