mlsanityData to 7 Oct 2026

Stochastic Runge-Kutta Methods: Provable Acceleration of Diffusion Models

arXiv 2410.04760 ↗First posted 7 Oct 2024stat.ML cs.LGcited by 35

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Diffusion models play a pivotal role in contemporary generative modeling, claiming state-of-the-art performance across various domains. Despite their superior sample quality, mainstream diffusion-based stochastic samplers like DDPM often require a large number of score function evaluations, incurring considerably higher computational cost compared to single-step generators like generative adversarial networks. While several acceleration methods have been proposed in practice, the theoretical foundations for accelerating diffusion models remain underexplored. In this paper, we propose and analyze a training-free acceleration algorithm for SDE-style diffusion samplers, based on the stochastic Runge-Kutta method. The proposed sampler provably attains varepsilon2 error -- measured in KL divergence -- using widetilde O(d3/2 / varepsilon) score function evaluations (for sufficiently small varepsilon), strengthening the state-of-the-art guarantees widetilde O(d3 / varepsilon) in terms of dimensional dependency. Numerical experiments validate the efficiency of the proposed method.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 72 of its 86 references that are in the map. First posted on arXiv 7 Oct 2024.

Builds on

72 of 86 references.

Closest ancestors · 52

Same direction, newest first

  1. 2024O(d/T) Convergence Theory for Diffusion Probabilistic Models under Minimal Assumptions27 Sep
  2. 2024What does guidance do? A fine-grained analysis in a simple setting19 Sep
  3. 2024From optimal score matching to optimal sampling11 Sep
  4. 2024A Sharp Convergence Theory for The Probability Flow ODEs of Diffusion Models5 Aug
  5. 2024Faster Diffusion Sampling with Randomized Midpoints: Sequential and Parallel3 Jun
  6. 2024Adapting to Unknown Low-Dimensional Structures in Score-Based Diffusion Models23 May
  7. 2024Learning general Gaussian mixtures with efficient score matching29 Apr
  8. 2024Convergence Analysis of Probability Flow ODE for Score-based Generative Models15 Apr
  9. 2024An Overview of Diffusion Models: Applications, Guided Generation, Statistical Rates and Optimization11 Apr
  10. 2024Accelerating Convergence of Score-Based Diffusion Models, Provably6 Mar
  11. 2024Theoretical Insights for Diffusion Guidance: A Case Study for Gaussian Mixture Models3 Mar
  12. 2024Minimax Optimality of Score-based Diffusion Models: Beyond the Density Lower Bound Assumptions23 Feb
38 more
  1. 2024Towards a mathematical theory for consistency training in diffusion models12 Feb
  2. 2024Score-based Diffusion Models via Stochastic Differential Equations -- a Technical Tutorial12 Feb
  3. 2024Convergence Analysis for General Probability Flow ODEs of Diffusion Models in Wasserstein Distances31 Jan
  4. 2024Neural Network-Based Score Estimation in Diffusion Models: Optimization and Generalization28 Jan
  5. 2023DPM-Solver-v3: Improved Diffusion ODE Solver with Empirical Model Statistics20 Oct
  6. 2023Deep Networks as Denoising Algorithms: Sample-Efficient Learning of Diffusion Models in High-Dimensional Graphical Models20 Sep
  7. 2023SA-Solver: Stochastic Adams Solver for Fast Sampling of Diffusion Models10 Sep
  8. 2023Nearly d-Linear Convergence Bounds for Diffusion Models via Stochastic Localization7 Aug
  9. 2023Towards Faster Non-Asymptotic Convergence for Diffusion-Based Generative Models15 Jun
  10. 2023Error Bounds for Flow Matching Methods26 May
  11. 2023SEEDS: Exponential SDE Solvers for Fast High-Quality Sampling from Diffusion Models23 May
  12. 2023The probability flow ODE is provably fast19 May
  13. 2023Restoration-Degradation Beyond Linear Diffusions: A Non-Asymptotic Analysis For DDIM-Type Samplers6 Mar
  14. 2023Diffusion Models are Minimax Optimal Distribution Estimators3 Mar
  15. 2023Consistency Models2 Mar
  16. 2023Score Approximation, Estimation and Distribution Recovery of Diffusion Models on Low-Dimensional Data14 Feb
  17. 2023UniPC: A Unified Predictor-Corrector Framework for Fast Sampling of Diffusion Models9 Feb
  18. 2022Improved Analysis of Score-based Generative Modeling: User-Friendly Bounds under Minimal Smoothness Assumptions3 Nov
  19. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models2 Nov
  20. 2022On Distillation of Guided Diffusion Models6 Oct
  21. 2022Convergence of score-based generative modeling for general data distributions26 Sep
  22. 2022Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions22 Sep
  23. 2022Diffusion Models: A Comprehensive Survey of Methods and Applications2 Sep
  24. 2022Classifier-Free Diffusion Guidance26 Jul
  25. 2022Maximum Likelihood Training for Score-Based Diffusion ODEs by High-Order Denoising Score Matching16 Jun
  26. 2022Convergence for score-based generative modeling with polynomial complexity13 Jun
  27. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps2 Jun
  28. 2022Elucidating the Design Space of Diffusion-Based Generative Models1 Jun
  29. 2022Fast Sampling of Diffusion Models with Exponential Integrator29 Apr
  30. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  31. 2021Structured Denoising Diffusion Models in Discrete State-Spaces7 Jul
  32. 2021Cascaded Diffusion Models for High Fidelity Image Generation30 May
  33. 2021Gotta Go Fast When Generating Data with Score-Based Models28 May
  34. 2021Diffusion Models Beat GANs on Image Synthesis11 May
  35. 2021Noise Estimation for Generative Diffusion Models6 Apr
  36. 2021Improved Denoising Diffusion Probabilistic Models18 Feb
  37. 2021Knowledge Distillation in Iterative Generative Models for Improved Sampling Speed7 Jan
  38. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov

50 newest of 52 All 52 (JSON)

Other references · 20

  1. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  2. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  3. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  4. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  5. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  6. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  7. 2021CSDI: Conditional Score-based Diffusion Models for Probabilistic Time Series Imputationcited by 577
  8. 2022GeoDiff: a Geometric Diffusion Model for Molecular Conformation Generationcited by 511
  9. 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speechcited by 445
  10. 2017A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasetscited by 374
  11. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 307
  12. 2023Imitating Human Behaviour with Diffusion Modelscited by 260
8 more
  1. 2022Diffusion probabilistic modeling of protein backbones in 3D for the motif-scaffolding problemcited by 229
  2. 2022Diffusion-based Time Series Imputation and Forecasting with Structured State Space Modelscited by 195
  3. 2020Generative Modeling with Denoising Auto-Encoders and Langevin Samplingcited by 118
  4. 2022Protein Structure and Sequence Generation with Equivariant Denoising Diffusion Probabilistic Modelscited by 97
  5. 2022Statistical Efficiency of Score Matching: The View from Isoperimetrycited by 63
  6. 2024Optimal score estimation via empirical Bayes smoothingcited by 60
  7. 2022Guided-TTS 2: A Diffusion Model for High-quality Adaptive Text-to-Speech with Untranscribed Datacited by 35
  8. 2024Optimal convex M-estimation via score matchingcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 18; 2026: 13.

35 papers.

Sort:

By area:

  1. 2025Fast Solvers for Discrete Diffusion Models: Theory and Applications of High-Order Algorithmscited by 47
  2. 2024Denoising diffusion probabilistic models are optimally adaptive to unknown low dimensionalitycited by 44
  3. 2024Understanding Generalizability of Diffusion Models Requires Rethinking the Hidden Gaussian Structurecited by 38
  4. 2025Low-dimensional adaptation of diffusion models: Convergence in total variationcited by 38
  5. 2024Provable Acceleration for Diffusion Models under Minimal Assumptionscited by 27
  6. 2025Minimax Optimality of the Probability Flow ODE for Diffusion Modelscited by 23
  7. 2025Dimension-Free Convergence of Diffusion Models for Approximate Gaussian Mixturescited by 23
  8. 2024On Statistical Rates of Conditional Diffusion Transformers: Approximation, Estimation and Minimax Optimalitycited by 21
  9. 2025Solving Inverse Problems via Diffusion-Based Priors: An Approximation-Free Ensemble Sampling Approachcited by 18
  10. 2025Advancing Wasserstein Convergence Analysis of Score-Based Models: Insights from Discretization and Second-Order Accelerationcited by 17
  11. 2025Faster Diffusion Models via Higher-Order Approximationcited by 16
  12. 2025Optimal Convergence Analysis of DDPM for General Distributionscited by 16
23 more
  1. 2025Lipschitz-Guided Design of Interpolation Schedules in Generative Modelscited by 15
  2. 2026Conditional Diffusion Guidance under Hard Constraint: A Stochastic Analysis Approachcited by 10
  3. 2025Fast Convergence for High-Order ODE Solvers in Diffusion Probabilistic Modelscited by 9
  4. 2025Assessing the Quality of Denoising Diffusion Models in Wasserstein Distance: Noisy Score and Optimal Boundscited by 8
  5. 2025Sublinear iterations can suffice even for DDPMscited by 7
  6. 2025Towards a unified framework for guided diffusion modelscited by 7
  7. 2026High-accuracy and dimension-free sampling with diffusionscited by 7
  8. 2025Information-Theoretic Proofs for Diffusion Samplingcited by 6
  9. 2026Diffusion Models with Heavy-Tailed Targets: Score Estimation and Sampling Guaranteescited by 4
  10. 2025Beyond Scores: Proximal Diffusion Modelscited by 3
  11. 2026ART for Diffusion Sampling: A Reinforcement Learning Approach to Timestep Schedulecited by 3
  12. 2025Deep Neural Networks Inspired by Differential Equationscited by 2
  13. 2026Query Lower Bounds for Diffusion Samplingcited by 2
  14. 2025ProxT2I: Efficient Reward-Guided Text-to-Image Generation via Proximal Diffusioncited by 1
  15. 2026Provably Reliable Classifier Guidance via Cross-Entropy Controlcited by 1
  16. 2026On the Robustness of Distribution Support under Diffusion Guidancecited by 1
  17. 2026From Scores to Gibbs Correctors: Accelerating Uniform-Rate Discrete Diffusion Modelscited by 1
  18. 2026Colored Noise Diffusion Samplingcited by 1
  19. 2026Diffusion Models for Adaptive Sequential Data Generationcited by 1
  20. 2026Diffusion Models Adapt to Low-Dimensional Structure Under Flexible Coefficient Choicescited by 1
  21. 2025Are First-Order Diffusion Samplers Really Slower? A Fast Forward-Value Approachcited by 0
  22. 2026Recursively Trained Diffusion Models: Limiting Collapse Distribution and Spectral Characterizationcited by 0
  23. 2026ART for Diffusion Sampling: Continuous-Time Control and Actor-Critic Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Convergence of Deterministic and Stochastic Diffusion-Model Samplers: A Simple Analysis in Wasserstein Distance
  2. 2022Accelerating Diffusion Models via Early Stop of the Diffusion Process
  3. 2025Pretrained Diffusion Models Are Inherently Skipped-Step Samplers
  4. 2023A Geometric Perspective on Diffusion Models
  5. 2025From Score Matching to Diffusion: A Fine-Grained Error Analysis in the Gaussian Setting
  6. 2023Denoising Diffusion Samplers
  7. 2025Diffusion Models: A Mathematical Introduction
  8. 2022Learning Fast Samplers for Diffusion Models by Differentiating Through Sample Quality