mlsanityData to 7 Oct 2026

Towards a mathematical theory for consistency training in diffusion models

arXiv 2402.07802 ↗First posted 12 Feb 2024stat.ML cs.IT cs.LGcited by 25

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Consistency models, which were proposed to mitigate the high computational overhead during the sampling phase of diffusion models, facilitate single-step sampling while attaining state-of-the-art empirical performance. When integrated into the training phase, consistency models attempt to train a sequence of consistency functions capable of mapping any point at any time step of the diffusion process to its starting point. Despite the empirical success, a comprehensive theoretical understanding of consistency training remains elusive. This paper takes a first step towards establishing theoretical underpinnings for consistency models. We demonstrate that, in order to generate samples within varepsilon proximity to the target in distribution (measured by some Wasserstein metric), it suffices for the number of steps in consistency learning to exceed the order of d5/2/varepsilon, with d the data dimension. Our theory offers rigorous insights into the validity and efficacy of consistency models, illuminating their utility in downstream inference tasks.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 32 of its 41 references that are in the map. First posted on arXiv 12 Feb 2024.

Builds on

32 of 41 references.

Closest ancestors · 23

Same direction, newest first

  1. 2023Improved Techniques for Training Consistency Models22 Oct
  2. 2023Consistency Trajectory Models: Learning Probability Flow ODE Trajectory of Diffusion1 Oct
  3. 2023SA-Solver: Stochastic Adams Solver for Fast Sampling of Diffusion Models10 Sep
  4. 2023Towards Faster Non-Asymptotic Convergence for Diffusion-Based Generative Models15 Jun
  5. 2023The probability flow ODE is provably fast19 May
  6. 2023Restoration-Degradation Beyond Linear Diffusions: A Non-Asymptotic Analysis For DDIM-Type Samplers6 Mar
  7. 2023Consistency Models2 Mar
  8. 2022Accelerating Diffusion Sampling with Classifier-based Feature Distillation22 Nov
  9. 2022Improved Analysis of Score-based Generative Modeling: User-Friendly Bounds under Minimal Smoothness Assumptions3 Nov
  10. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models2 Nov
  11. 2022Convergence of score-based generative modeling for general data distributions26 Sep
  12. 2022Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions22 Sep
11 more
  1. 2022Let us Build Bridges: Understanding and Extending Diffusion Generative Models31 Aug
  2. 2022Convergence of denoising diffusion models under the manifold hypothesis10 Aug
  3. 2022Score-Based Generative Models Detect Manifolds2 Jun
  4. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps2 Jun
  5. 2022Elucidating the Design Space of Diffusion-Based Generative Models1 Jun
  6. 2022Fast Sampling of Diffusion Models with Exponential Integrator29 Apr
  7. 2021Diffusion Schr"odinger Bridge with Applications to Score-Based Generative Modeling1 Jun
  8. 2021Diffusion Models Beat GANs on Image Synthesis11 May
  9. 2021Knowledge Distillation in Iterative Generative Models for Improved Sampling Speed7 Jan
  10. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  11. 2020Improved Techniques for Training Score-Based Generative Models16 Jun

Other references · 9

  1. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  2. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  3. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  4. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  5. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,339
  6. 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speechcited by 445
  7. 2020Generative Modeling with Denoising Auto-Encoders and Langevin Samplingcited by 118
  8. 2023Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learningcited by 93
  9. 2023VideoLCM: Video Latent Consistency Modelcited by 90

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 12; 2026: 2.

25 papers.

Sort:

By area:

  1. 2025Inference-Time Alignment in Diffusion Models with Reward-Guided Generation: Tutorial and Reviewcited by 82
  2. 2024Accelerating Convergence of Score-Based Diffusion Models, Provablycited by 74
  3. 2024Provably Robust Score-Based Diffusion Posterior Sampling for Plug-and-Play Image Reconstructioncited by 73
  4. 2024A Sharp Convergence Theory for The Probability Flow ODEs of Diffusion Modelscited by 64
  5. 2024Theoretical Insights for Diffusion Guidance: A Case Study for Gaussian Mixture Modelscited by 46
  6. 2024Denoising diffusion probabilistic models are optimally adaptive to unknown low dimensionalitycited by 44
  7. 2024Critical windows: non-asymptotic theory for feature emergence in diffusion modelscited by 41
  8. 2025Low-dimensional adaptation of diffusion models: Convergence in total variationcited by 38
  9. 2024Stochastic Runge-Kutta Methods: Provable Acceleration of Diffusion Modelscited by 35
  10. 2024Broadening Target Distributions for Accelerated Diffusion Models via a Novel Analysis Approachcited by 19
  11. 2024On the Feature Learning in Diffusion Modelscited by 19
  12. 2025Faster Diffusion Models via Higher-Order Approximationcited by 16
13 more
  1. 2024On the Convergence and Straightness of Rectified Flowcited by 11
  2. 2025ReAlign: Text-to-Motion Generation via Step-Aware Reward-Guided Alignmentcited by 10
  3. 2025A Sharp KL-Convergence Analysis for Diffusion Models under Minimal Assumptionscited by 8
  4. 2025MIRA: Towards Mitigating Reward Hacking in Inference-Time Alignment of T2I Diffusion Modelscited by 7
  5. 2024Theory on Score-Mismatched Diffusion Models and Zero-Shot Conditional Samplerscited by 6
  6. 2025Convergence Of Consistency Model With Multistep Sampling Under General Data Assumptionscited by 4
  7. 2025Elucidating Rectified Flow with Deterministic Sampler: Polynomial Discretization Complexity for Multi and One-step Modelscited by 3
  8. 2025Computational bottlenecks for denoising diffusionscited by 2
  9. 2025Multi-Step Consistency Models: Fast Generation with Theoretical Guaranteescited by 2
  10. 2026Low-Dimensional Adaptation of Rectified Flow: A Diffusion and Stochastic Localization Perspectivecited by 2
  11. 2025Non-asymptotic convergence bound of conditional diffusion modelscited by 1
  12. 2025Continuous Semi-Implicit Modelscited by 0
  13. 2026On the Memorization of Consistency Distillation for Diffusion Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023On the Equivalence of Consistency-Type Models: Consistency Models, Consistent Diffusion Models, and Fokker-Planck Regularization
  2. 2023Consistent Diffusion Models: Mitigating Sampling Drift by Learning to be Consistent
  3. 2024Truncated Consistency Models
  4. 2023The Emergence of Reproducibility and Generalizability in Diffusion Models
  5. 2026Consistent Diffusion Language Models
  6. 2024Provable Statistical Rates for Consistency Diffusion Models
  7. 2024Towards Faster Training of Diffusion Models: An Inspiration of A Consistency Phenomenon
  8. 2024Consistency Models Made Easy