mlsanityData to 13 Oct 2026

Distilling Diffusion Models into Conditional GANs

arXiv 2405.05967 ↗First posted 9 May 2024cs.CV cs.GR cs.LGcited by 78

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

We propose a method to distill a complex multistep diffusion model into a single-step conditional GAN student model, dramatically accelerating inference, while preserving image quality. Our approach interprets diffusion distillation as a paired image-to-image translation task, using noise-to-image pairs of the diffusion model's ODE trajectory. For efficient regression loss computation, we propose E-LatentLPIPS, a perceptual loss operating directly in diffusion model's latent space, utilizing an ensemble of augmentations. Furthermore, we adapt a diffusion model to construct a multi-scale discriminator with a text alignment loss to build an effective conditional GAN-based formulation. E-LatentLPIPS converges more efficiently than many existing distillation methods, even accounting for dataset construction costs. We demonstrate that our one-step generator outperforms cutting-edge one-step diffusion distillation models -- DMD, SDXL-Turbo, and SDXL-Lightning -- on the zero-shot COCO benchmark.

Key facts. Cited by 78 papers in the map (data through 13 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 94 of its 107 references that are in the map. First posted on arXiv 9 May 2024.

Builds on

94 of 107 references.

Closest ancestors · 26

Same direction, newest first

  1. 2024Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion Distillation18 Mar
  2. 2024SDXL-Lightning: Progressive Adversarial Diffusion Distillation21 Feb
  3. 2023Analyzing and Improving the Training Dynamics of Diffusion Models5 Dec
  4. 2023One-step Diffusion with Distribution Matching Distillation30 Nov
  5. 2023Adversarial Diffusion Distillation28 Nov
  6. 2023LCM-LoRA: A Universal Stable-Diffusion Acceleration Module9 Nov
  7. 2023Improved Techniques for Training Consistency Models22 Oct
  8. 2023Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference6 Oct
  9. 2023Consistency Trajectory Models: Learning Probability Flow ODE Trajectory of Diffusion1 Oct
  10. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis4 Jul
  11. 2023BOOT: Data-free Distillation of Denoising Diffusion Models with Bootstrapping8 Jun
  12. 2023Speed Is All You Need: On-Device Acceleration of Large Diffusion Models via GPU-Aware Optimizations21 Apr
14 more
  1. 2023TRACT: Denoising Diffusion Models with Transitive Closure Time-Distillation7 Mar
  2. 2023Consistency Models2 Mar
  3. 2022Fast Sampling of Diffusion Models via Operator Learning24 Nov
  4. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models2 Nov
  5. 2022On Distillation of Guided Diffusion Models6 Oct
  6. 2022Classifier-Free Diffusion Guidance26 Jul
  7. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps2 Jun
  8. 2022Elucidating the Design Space of Diffusion-Based Generative Models1 Jun
  9. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  10. 2021Tackling the Generative Learning Trilemma with Denoising Diffusion GANs15 Dec
  11. 2021Knowledge Distillation in Iterative Generative Models for Improved Sampling Speed7 Jan
  12. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  13. 2020Denoising Diffusion Implicit Models6 Oct
  14. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 68

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,093
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,223
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,550
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,321
  5. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,082
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,180
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,023
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,564
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,339
  10. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,000
  11. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,484
  12. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,516
38 more
  1. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,141
  2. 2013Auto-Encoding Variational Bayescited by 4,024
  3. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,607
  4. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,585
  5. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,294
  6. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,019
  7. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,687
  8. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,619
  9. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,604
  10. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,550
  11. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,421
  12. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,407
  13. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,240
  14. 2014Conditional Generative Adversarial Netscited by 2,157
  15. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,921
  16. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,768
  17. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,627
  18. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,422
  19. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,342
  20. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,296
  21. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,289
  22. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,288
  23. 2020Training Generative Adversarial Networks with Limited Datacited by 1,184
  24. 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,178
  25. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,104
  26. 2017StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translationcited by 1,036
  27. 2019Adversarial Examples Are Not Bugs, They Are Featurescited by 953
  28. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 928
  29. 2018Multimodal Unsupervised Image-to-Image Translationcited by 795
  30. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 769
  31. 2016Generative Adversarial Text to Image Synthesiscited by 735
  32. 2020Encoding in Style: a StyleGAN Encoder for Image-to-Image Translationcited by 727
  33. 2018Which Training Methods for GANs do actually Converge?cited by 676
  34. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  35. 2016Conditional Image Synthesis With Auxiliary Classifier GANscited by 563
  36. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
  37. 2022StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasetscited by 490
  38. 2018Assessing Generative Models via Precision and Recallcited by 405

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 19; 2025: 35; 2026: 24.

78 papers.

Sort:

By area:

  1. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Thinkcited by 679
  2. 2024From Slow Bidirectional to Fast Autoregressive Video Diffusion Modelscited by 414
  3. 2024SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Modelscited by 165
  4. 2025WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modelingcited by 158
  5. 2025Diffusion Adversarial Post-Training for One-Step Video Generationcited by 147
  6. 2025SANA-Sprint: One-Step Diffusion with Continuous-Time Consistency Distillationcited by 86
  7. 2025Autoregressive Adversarial Post-Training for Real-Time Interactive Video Generationcited by 86
  8. 2024Phased Consistency Modelscited by 85
  9. 2025What matters for Representation Alignment: Global Information or Spatial Structure?cited by 79
  10. 2025Learning Few-Step Diffusion Models by Trajectory Distribution Matchingcited by 52
  11. 2024Adversarial Diffusion Compression for Real-World Image Super-Resolutioncited by 51
  12. 2025SeedVR2: One-Step Video Restoration via Diffusion Adversarial Post-Trainingcited by 45
38 more
  1. 2024FLOAT: Generative Motion Latent Flow Matching for Audio-driven Talking Portraitcited by 40
  2. 2024InstantDrag: Improving Interactivity in Drag-based Image Editingcited by 30
  3. 2024LVCD: Reference-based Lineart Video Colorization with Diffusion Modelscited by 27
  4. 2025One-Step Diffusion-Based Image Compression with Semantic Distillationcited by 23
  5. 2024Layer- and Timestep-Adaptive Differentiable Token Compression Ratios for Efficient Diffusion Transformerscited by 21
  6. 2025Dual-Expert Consistency Model for Efficient and High-Quality Video Generationcited by 21
  7. 2024Boosting Latent Diffusion with Perceptual Objectivescited by 20
  8. 2025CMT: Mid-Training for Efficient Learning of Consistency, Mean Flow, and Flow Map Modelscited by 18
  9. 2024Diffusion Model for Planning: A Systematic Literature Reviewcited by 17
  10. 2024Presto! Distilling Steps and Layers for Accelerating Music Generationcited by 16
  11. 2025Exploring Multimodal Diffusion Transformers for Enhanced Prompt-based Image Editingcited by 16
  12. 2024Fast and Memory-Efficient Video Diffusion Using Streamlined Inferencecited by 14
  13. 2025Fast Text-to-Audio Generation with Adversarial Post-Trainingcited by 14
  14. 20242.5D Multi-view Averaging Diffusion Model for 3D Medical Image Translation: Application to Low-count PET Reconstruction with CT-less Attenuation Correctioncited by 13
  15. 2024InstantRestore: Single-Step Personalized Face Restoration with Shared-Image Attentioncited by 13
  16. 2025Improved Training Technique for Latent Consistency Modelscited by 13
  17. 2025Learning an Image Editing Model without Image Editing Pairscited by 13
  18. 2025RayFlow: Instance-Aware Diffusion Acceleration via Adaptive Flow Trajectoriescited by 12
  19. 2025Scaling Group Inference for Diverse and High-Quality Generationcited by 12
  20. 2026Gamma-World: Generative Multi-Agent World Modeling Beyond Two Playerscited by 12
  21. 2025Few-Step Diffusion via Score identity Distillationcited by 11
  22. 2025BWCache: Accelerating Video Diffusion Transformers through Block-Wise Cachingcited by 11
  23. 2025Adversarial Flow Modelscited by 11
  24. 2026PixelGen: Improving Pixel Diffusion with Perceptual Supervisioncited by 11
  25. 2024SoundCTM: Unifying Score-based and Consistency Models for Full-band Text-to-Sound Generationcited by 9
  26. 2025Autoregressive Distillation of Diffusion Transformerscited by 9
  27. 2025Fast Data Attribution for Text-to-Image Modelscited by 7
  28. 2026Adaptive Video Distillation: Mitigating Oversaturation and Temporal Collapse in Few-Step Generationcited by 7
  29. 2026Stable Audio 3cited by 6
  30. 2024Timestep-Aware Diffusion Model for Extreme Image Rescalingcited by 5
  31. 2025Balanced Image Stylization with Style Matching Scorecited by 5
  32. 2025Differentiable Solver Search for Fast Diffusion Samplingcited by 5
  33. 2025Revisiting Diffusion Models: From Generative Pre-training to One-Step Generationcited by 5
  34. 2025ShortFT: Diffusion Model Alignment via Shortcut-based Fine-Tuningcited by 5
  35. 2024TLCM: Training-efficient Latent Consistency Model for Image Generation with 2-8 Stepscited by 4
  36. 2025Distilling Multi-view Diffusion Models into 3D Generatorscited by 4
  37. 2026DUO-VSR: Dual-Stream Distillation for One-Step Video Super-Resolutioncited by 4
  38. 2026Continuous Adversarial Flow Modelscited by 4

50 most cited of 78 All 78 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Uni-DAD: Unified Distillation and Adaptation of Diffusion Models for Few-step Few-shot Image Generation
  2. 2024One-Step Image Translation with Text-to-Image Models
  3. 2024Unlocking Dataset Distillation with Diffusion Models
  4. 2024Latent Denoising Diffusion GAN: Faster sampling, Higher image quality
  5. 2023Conditional Image Generation with Pretrained Generative Model
  6. 2021Diffusion Models Beat GANs on Image Synthesis
  7. 2024E2GAN: Efficient Training of Efficient GANs for Image-to-Image Translation
  8. 2023Diffusion Models Beat GANs on Image Classification