mlsanityData to 14 Oct 2026

Improved Techniques for Training Consistency Models

arXiv 2310.14189 ↗First posted 22 Oct 2023cs.LGcited by 439

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Consistency models are a nascent family of generative models that can sample high quality data in one step without the need for adversarial training. Current consistency models achieve optimal sample quality by distilling from pre-trained diffusion models and employing learned metrics such as LPIPS. However, distillation limits the quality of consistency models to that of the pre-trained diffusion model, and LPIPS causes undesirable bias in evaluation. To tackle these challenges, we present improved techniques for consistency training, where consistency models learn directly from data without distillation. We delve into the theory behind consistency training and identify a previously overlooked flaw, which we address by eliminating Exponential Moving Average from the teacher consistency model. To replace learned metrics like LPIPS, we adopt Pseudo-Huber losses from robust statistics. Additionally, we introduce a lognormal noise schedule for the consistency training objective, and propose to double total discretization steps every set number of training iterations. Combined with better hyperparameter tuning, these modifications enable consistency models to achieve FID scores of 2.51 and 3.25 on CIFAR-10 and ImageNet 64× 64 respectively in a single sampling step. These scores mark a 3.5× and 4× improvement compared to prior consistency training approaches. Through two-step sampling, we further reduce FID scores to 2.24 and 2.77 on these two datasets, surpassing those obtained via distillation in both one-step and two-step settings, while narrowing the gap between consistency models and other state-of-the-art generative models.

Key facts. Cited by 439 papers in the map (data through 14 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 41 of its 48 references that are in the map. First posted on arXiv 22 Oct 2023.

Builds on

41 of 48 references.

Closest ancestors · 19

Same direction, newest first

  1. 2023BOOT: Data-free Distillation of Denoising Diffusion Models with Bootstrapping8 Jun
  2. 2023Diff-Instruct: A Universal Approach for Transferring Knowledge From Pre-trained Diffusion Models29 May
  3. 2023TRACT: Denoising Diffusion Models with Transitive Closure Time-Distillation7 Mar
  4. 2023Consistency Models2 Mar
  5. 2023UniPC: A Unified Predictor-Corrector Framework for Fast Sampling of Diffusion Models9 Feb
  6. 2023Simple diffusion: End-to-end diffusion for high resolution images26 Jan
  7. 2022Refining Generative Process with Discriminator Guidance in Score-based Diffusion Models28 Nov
  8. 2022Fast Sampling of Diffusion Models via Operator Learning24 Nov
  9. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps2 Jun
  10. 2022Elucidating the Design Space of Diffusion-Based Generative Models1 Jun
  11. 2022Fast Sampling of Diffusion Models with Exponential Integrator29 Apr
  12. 2021Score-based Generative Modeling in Latent Space10 Jun
7 more
  1. 2021Diffusion Models Beat GANs on Image Synthesis11 May
  2. 2021Improved Denoising Diffusion Probabilistic Models18 Feb
  3. 2021Knowledge Distillation in Iterative Generative Models for Improved Sampling Speed7 Jan
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  5. 2020Denoising Diffusion Implicit Models6 Oct
  6. 2020Denoising Diffusion Probabilistic Models19 Jun
  7. 2020Improved Techniques for Training Score-Based Generative Models16 Jun

Other references · 22

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  3. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  4. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  5. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  6. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,588
  7. 2016Improved Techniques for Training GANscited by 2,970
  8. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,626
  9. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,200
  10. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,545
  11. 2020Training Generative Adversarial Networks with Limited Datacited by 1,186
  12. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 944
10 more
  1. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 772
  2. 2020NVAE: A Deep Hierarchical Variational Autoencodercited by 637
  3. 2022StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasetscited by 492
  4. 2019Sliced Score Matching: A Scalable Approach to Density and Score Estimationcited by 384
  5. 2019Residual Flows for Invertible Generative Modelingcited by 257
  6. 2022The Role of ImageNet Classes in Fr'echet Inception Distancecited by 213
  7. 2021Projected GANs Converge Fastercited by 208
  8. 2022Scalable Adaptive Computation for Iterative Generationcited by 195
  9. 2021Towards Faster and Stabilized GAN Training for High-fidelity Few-shot Image Synthesiscited by 140
  10. 2022Poisson Flow Generative Modelscited by 95

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 7; 2024: 104; 2025: 165; 2026: 163.

439 papers.

Sort:

By area:

  1. 2024Improved Distribution Matching Distillation for Fast Image Synthesiscited by 684
  2. 2025Mean Flows for One-step Generative Modelingcited by 561
  3. 2024From Slow Bidirectional to Fast Autoregressive Video Diffusion Modelscited by 433
  4. 2024One Step Diffusion via Shortcut Modelscited by 391
  5. 2024Simplifying, Stabilizing and Scaling Continuous-Time Consistency Modelscited by 283
  6. 2024Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion Distillationcited by 279
  7. 2024SDXL-Lightning: Progressive Adversarial Diffusion Distillationcited by 266
  8. 2024Score identity Distillation: Exponentially Fast Distillation of Pretrained Diffusion Models for One-Step Generationcited by 214
  9. 2025Unified Video Action Modelcited by 205
  10. 2026Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generationcited by 184
  11. 2024Consistency Policy: Accelerated Visuomotor Policies via Consistency Distillationcited by 166
  12. 2025Diffusion Adversarial Post-Training for One-Step Video Generationcited by 151
38 more
  1. 2024Consistency Models Made Easycited by 144
  2. 2025Training Agents Inside of Scalable World Modelscited by 136
  3. 2025The Diffusion Dualitycited by 135
  4. 2026Generative Modeling via Driftingcited by 116
  5. 2025Inductive Moment Matchingcited by 113
  6. 2025Improved Mean Flows: On the Challenges of Fastforward Generative Modelscited by 110
  7. 2024Flow map matching with stochastic interpolants: A mathematical framework for consistency modelscited by 109
  8. 2024Improving the Training of Rectified Flowscited by 108
  9. 2024Consistency Flow Matching: Defining Straight Flows with Velocity Consistencycited by 104
  10. 2025How to build a consistency model: Learning flow maps via self-distillationcited by 103
  11. 2024Normalizing Flows are Capable Generative Modelscited by 91
  12. 2025Autoregressive Adversarial Post-Training for Real-Time Interactive Video Generationcited by 90
  13. 2024T2V-Turbo: Breaking the Quality Bottleneck of Video Consistency Model with Mixed Reward Feedbackcited by 87
  14. 2025Align Your Flow: Scaling Continuous-Time Flow Map Distillationcited by 87
  15. 2025Large Scale Diffusion Distillation via Score-Regularized Continuous-Time Consistencycited by 86
  16. 2024Phased Consistency Modelscited by 85
  17. 2024Multistep Distillation of Diffusion Models via Moment Matchingcited by 82
  18. 2024The Matrix: Infinite-Horizon World Generation with Real-Time Moving Controlcited by 80
  19. 2025Scaling Instruction-Based Video Editing with a High-Quality Synthetic Datasetcited by 79
  20. 2024Distilling Diffusion Models into Conditional GANscited by 78
  21. 2025Marigold: Affordable Adaptation of Diffusion-Based Image Generators for Image Analysiscited by 78
  22. 2023Diffusion Models Without Attentioncited by 75
  23. 2024CLLMs: Consistency Large Language Modelscited by 72
  24. 2024One-Step Diffusion Distillation through Score Implicit Matchingcited by 71
  25. 2025Diffuse and Disperse: Image Generation with Representation Regularizationcited by 70
  26. 2024Multistep Consistency Modelscited by 67
  27. 2024Score-based Diffusion Models via Stochastic Differential Equations -- a Technical Tutorialcited by 60
  28. 2024ControlVAR: Exploring Controllable Visual Autoregressive Modelingcited by 58
  29. 2025Learning Few-Step Diffusion Models by Trajectory Distribution Matchingcited by 57
  30. 2024Flash Diffusion: Accelerating Any Conditional Diffusion Model for Few Steps Image Generationcited by 56
  31. 2026Causal Forcing++: Scalable Few-Step Autoregressive Diffusion Distillation for Real-Time Interactive Video Generationcited by 56
  32. 2024Sora as a World Model? A Complete Survey on Text-to-Video Generationcited by 53
  33. 2026One-step Latent-free Image Generation with Pixel Mean Flowscited by 53
  34. 2025SeedVR2: One-Step Video Restoration via Diffusion Adversarial Post-Trainingcited by 50
  35. 2024SD-DiT: Unleashing the Power of Self-supervised Discrimination in Diffusion Transformercited by 49
  36. 2024Distillation of Discrete Diffusion through Dimensional Correlationscited by 48
  37. 2024AnimateDiff-Lightning: Cross-Model Diffusion Distillationcited by 47
  38. 2024Adversarial Score identity Distillation: Rapidly Surpassing the Teacher in One Stepcited by 47

50 most cited of 439 All 439 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Enhancing Consistency-Based Image Generation via Adversarialy-Trained Classification and Energy-Based Discrimination
  2. 2023ACT-Diffusion: Efficient Adversarial Consistency Training for One-step Diffusion Models
  3. 2024Provable Statistical Rates for Consistency Diffusion Models
  4. 2023Convergence guarantee for consistency models
  5. 2023Consistent Diffusion Models: Mitigating Sampling Drift by Learning to be Consistent
  6. 2026Consistency Training Along the Transformer Stack
  7. 2024Towards Faster Training of Diffusion Models: An Inspiration of A Consistency Phenomenon
  8. 2025Improved Diffusion-based Generative Model with Better Adversarial Robustness