mlsanityData to 7 Oct 2026

Score identity Distillation: Exponentially Fast Distillation of Pretrained Diffusion Models for One-Step Generation

arXiv 2404.04057 ↗First posted 5 Apr 2024cs.LG cs.AI cs.CVcited by 211

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

We introduce Score identity Distillation (SiD), an innovative data-free method that distills the generative capabilities of pretrained diffusion models into a single-step generator. SiD not only facilitates an exponentially fast reduction in Fr'echet inception distance (FID) during distillation but also approaches or even exceeds the FID performance of the original teacher diffusion models. By reformulating forward diffusion processes as semi-implicit distributions, we leverage three score-related identities to create an innovative loss mechanism. This mechanism achieves rapid FID reduction by training the generator using its own synthesized images, eliminating the need for real data or reverse-diffusion-based generation, all accomplished within significantly shortened generation time. Upon evaluation across four benchmark datasets, the SiD algorithm demonstrates high iteration efficiency during distillation and surpasses competing distillation approaches, whether they are one-step or few-step, data-free, or dependent on training data, in terms of generation quality. This achievement not only redefines the benchmarks for efficiency and effectiveness in diffusion distillation but also in the broader field of diffusion-based generation. The PyTorch implementation is available at https://github.com/mingyuanzhou/SiD

Key facts. Cited by 211 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 71 of its 88 references that are in the map. First posted on arXiv 5 Apr 2024.

Builds on

71 of 88 references.

Closest ancestors · 35

Same direction, newest first

  1. 2023One-Step Diffusion Distillation via Deep Equilibrium Models12 Dec
  2. 2023One-step Diffusion with Distribution Matching Distillation30 Nov
  3. 2023Adversarial Diffusion Distillation28 Nov
  4. 2023LCM-LoRA: A Universal Stable-Diffusion Acceleration Module9 Nov
  5. 2023Hierarchical Semi-Implicit Variational Inference with Application to Diffusion Model Acceleration26 Oct
  6. 2023Improved Techniques for Training Consistency Models22 Oct
  7. 2023Learning Stackable and Skippable LEGO Bricks for Efficient, Reconfigurable, and Variable-Resolution Diffusion Modeling10 Oct
  8. 2023Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference6 Oct
  9. 2023Consistency Trajectory Models: Learning Probability Flow ODE Trajectory of Diffusion1 Oct
  10. 2023Score Mismatching for Generative Modeling20 Sep
  11. 2023SA-Solver: Stochastic Adams Solver for Fast Sampling of Diffusion Models10 Sep
  12. 2023BOOT: Data-free Distillation of Denoising Diffusion Models with Bootstrapping8 Jun
23 more
  1. 2023Diff-Instruct: A Universal Approach for Transferring Knowledge From Pre-trained Diffusion Models29 May
  2. 2023Patch Diffusion: Faster and More Data-Efficient Training of Diffusion Models25 Apr
  3. 2023TRACT: Denoising Diffusion Models with Transitive Closure Time-Distillation7 Mar
  4. 2023Consistency Models2 Mar
  5. 2022Scalable Diffusion Models with Transformers19 Dec
  6. 2022Fast Sampling of Diffusion Models via Operator Learning24 Nov
  7. 2022On Distillation of Guided Diffusion Models6 Oct
  8. 2022Understanding Diffusion Models: A Unified Perspective25 Aug
  9. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps2 Jun
  10. 2022Elucidating the Design Space of Diffusion-Based Generative Models1 Jun
  11. 2022Accelerating Diffusion Models via Early Stop of the Diffusion Process25 May
  12. 2022Fast Sampling of Diffusion Models with Exponential Integrator29 Apr
  13. 2022Pseudo Numerical Methods for Diffusion Models on Manifolds20 Feb
  14. 2022DiffuseVAE: Efficient, Controllable and High-Fidelity Generation from Low-Dimensional Latents2 Jan
  15. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  16. 2021Tackling the Generative Learning Trilemma with Denoising Diffusion GANs15 Dec
  17. 2021Variational Diffusion Models1 Jul
  18. 2021Cascaded Diffusion Models for High Fidelity Image Generation30 May
  19. 2021Diffusion Models Beat GANs on Image Synthesis11 May
  20. 2021Improved Denoising Diffusion Probabilistic Models18 Feb
  21. 2021Knowledge Distillation in Iterative Generative Models for Improved Sampling Speed7 Jan
  22. 2020Denoising Diffusion Implicit Models6 Oct
  23. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 36

  1. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  2. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  3. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  4. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  5. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  6. 2022Flow Matching for Generative Modelingcited by 5,426
  7. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  8. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  9. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  10. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  11. 2016Improved Techniques for Training GANscited by 2,938
  12. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
24 more
  1. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillationcited by 1,207
  2. 2019StarGAN v2: Diverse Image Synthesis for Multiple Domainscited by 1,095
  3. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 925
  4. 2020NVAE: A Deep Hierarchical Variational Autoencodercited by 635
  5. 2022Improving Diffusion Models for Inverse Problems using Manifold Constraintscited by 501
  6. 2020Differentiable Augmentation for Data-Efficient GAN Trainingcited by 361
  7. 2021Projected GANs Converge Fastercited by 204
  8. 2022Diffusion-GAN: Training GANs with Diffusioncited by 203
  9. 2022Scalable Adaptive Computation for Iterative Generationcited by 190
  10. 2023UFOGen: You Forward Once Large Scale Text-to-Image Generation via Diffusion GANscited by 165
  11. 2020A Two-Timescale Framework for Bilevel Optimization: Complexity Analysis and Application to Actor-Criticcited by 141
  12. 2023SwiftBrush: One-Step Text-to-Image Diffusion Model with Variational Score Distillationcited by 109
  13. 2023On Penalty-based Bilevel Gradient Descent Methodcited by 87
  14. 2018Semi-Implicit Variational Inferencecited by 55
  15. 2019Semi-Implicit Graph Variational Auto-Encoderscited by 54
  16. 2017Assigning a value to a power likelihood in a general Bayesian modelcited by 46
  17. 2018Unbiased Implicit Variational Inferencecited by 33
  18. 2020Exploiting Chain Rule and Bayes' Theorem to Compare Probability Distributionscited by 22
  19. 2019Importance Weighted Hierarchical Variational Inferencecited by 20
  20. 2018Doubly Semi-Implicit Variational Inferencecited by 19
  21. 2023Semi-Implicit Variational Inference via Score Matchingcited by 19
  22. 2019Variational approximations using Fisher divergencecited by 12
  23. 2019Energy-Inspired Models: Learning with Sampler-Induced Distributionscited by 12
  24. 2021Efficient Semi-Implicit Variational Inferencecited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 34; 2025: 99; 2026: 78.

211 papers.

Sort:

By area:

  1. 2024Improved Distribution Matching Distillation for Fast Image Synthesiscited by 648
  2. 2025Mean Flows for One-step Generative Modelingcited by 524
  3. 2024Simplifying, Stabilizing and Scaling Continuous-Time Consistency Modelscited by 276
  4. 2024Consistency Models Made Easycited by 140
  5. 2025Inductive Moment Matchingcited by 107
  6. 2026Generative Modeling via Driftingcited by 98
  7. 2024PeRFlow: Piecewise Rectified Flow as Universal Plug-and-Play Acceleratorcited by 95
  8. 2025SANA-Sprint: One-Step Diffusion with Continuous-Time Consistency Distillationcited by 85
  9. 2024One-Step Diffusion Policy: Fast Visuomotor Policies via Diffusion Distillationcited by 83
  10. 2025Align Your Flow: Scaling Continuous-Time Flow Map Distillationcited by 81
  11. 2025Large Scale Diffusion Distillation via Score-Regularized Continuous-Time Consistencycited by 81
  12. 2025PixNerd: Pixel Neural Field Diffusioncited by 75
38 more
  1. 2024One-Step Diffusion Distillation through Score Implicit Matchingcited by 68
  2. 2024EM Distillation for One-step Diffusion Modelscited by 60
  3. 2025Kling-Omni Technical Reportcited by 60
  4. 2024Rectified Diffusion: Straightness Is Not Your Need in Rectified Flowcited by 59
  5. 2025DeCo: Frequency-Decoupled Pixel Diffusion for End-to-End Image Generationcited by 55
  6. 2024Discrete Copula Diffusioncited by 53
  7. 2025Learning Few-Step Diffusion Models by Trajectory Distribution Matchingcited by 49
  8. 2024Adversarial Score identity Distillation: Rapidly Surpassing the Teacher in One Stepcited by 47
  9. 2025End-to-End Training for Autoregressive Video Diffusion via Self-Resamplingcited by 45
  10. 2025One-step Diffusion Models with f-Divergence Distribution Matchingcited by 43
  11. 2024LinFusion: 1 GPU, 1 Minute, 16K Imagecited by 42
  12. 2024Diffusion Policies creating a Trust Region for Offline Reinforcement Learningcited by 41
  13. 2025AlphaFlow: Understanding and Improving MeanFlow Modelscited by 40
  14. 2025Adversarial Distribution Matching for Diffusion Distillation Towards Efficient Image and Video Synthesiscited by 39
  15. 2025Decoupled DMD: CFG Augmentation as the Spear, Distribution Matching as the Shieldcited by 38
  16. 2024OSV: One Step is Enough for High-Quality Image to Video Generationcited by 36
  17. 2026DiffusionOPD: A Unified Perspective of On-Policy Distillation in Diffusion Modelscited by 35
  18. 2024Autoregressive Diffusion Transformer for Text-to-Speech Synthesiscited by 34
  19. 2025Transition Models: Rethinking the Generative Learning Objectivecited by 32
  20. 2025Continuously Augmented Discrete Diffusion model for Categorical Generative Modelingcited by 32
  21. 2026Representation Fr'echet Loss for Visual Generationcited by 32
  22. 2025MoFlow: One-Step Flow Matching for Human Trajectory Forecasting via Implicit Maximum Likelihood Estimation based Distillationcited by 31
  23. 2024Diff-Instruct++: Training One-step Text-to-image Generator Model to Align with Human Preferencescited by 30
  24. 2026Transition Matching Distillation for Fast Video Generationcited by 27
  25. 2024Flow Generator Matchingcited by 26
  26. 2025Unleashing Vecset Diffusion Model for Fast Shape Generationcited by 25
  27. 2025pi-Flow: Policy-Based Few-Step Generation via Imitation Distillationcited by 25
  28. 2025StreamAvatar: Streaming Diffusion Models for Real-Time Interactive Human Avatarscited by 24
  29. 2026Causal-rCM: A Unified Teacher-Forcing and Self-Forcing Open Recipe for Autoregressive Diffusion Distillation in Streaming Video Generation and Interactive World Modelscited by 23
  30. 2024Score Forgetting Distillation: A Swift, Data-Free Method for Machine Unlearning in Diffusion Modelscited by 22
  31. 2024Truncated Consistency Modelscited by 21
  32. 2025Dual-Expert Consistency Model for Efficient and High-Quality Video Generationcited by 21
  33. 2025Phased DMD: Few-step Distribution Matching Distillation via Score Matching within Subintervalscited by 21
  34. 2026Diversity-Preserved Distribution Matching Distillation for Fast Visual Synthesiscited by 20
  35. 2024AlignIQL: Policy Alignment in Implicit Q-Learning through Constrained Optimizationcited by 19
  36. 2025Reward-Instruct: A Reward-Centric Approach to Fast Photo-Realistic Image Generationcited by 19
  37. 2025DLM-One: Diffusion Language Models for One-Step Sequence Generationcited by 19
  38. 2025CMT: Mid-Training for Efficient Learning of Consistency, Mean Flow, and Flow Map Modelscited by 18

50 most cited of 211 All 211 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Simple and Fast Distillation of Diffusion Models
  2. 2026EMPURPLE: A Free Lunch for Diffusion Distillation based on the Information Bottleneck
  3. 2026Consistent Distribution Matching for Data-Free Diffusion Distillation
  4. 2025From Structure to Detail: Hierarchical Distillation for Efficient Diffusion Model
  5. 2024SFDDM: Single-fold Distillation for Diffusion models
  6. 2025Dimathtt[M]O: Distilling Masked Diffusion Models into One-step Generator
  7. 2024DDIL: Diversity Enhancing Diffusion Distillation With Imitation Learning
  8. 2026From Score Learning to Discretized Sampling: An End-to-End Generalization Analysis of Diffusion Models