# Distilling Diffusion Models into Conditional GANs

arXiv: https://arxiv.org/abs/2405.05967 · HTML: https://mlsanity.com/paper/2405.05967/

## Key facts

- Cited by 78 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 94 of its 107 references that are in the map.
- First posted on arXiv 9 May 2024.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 30 of 94)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)

## Built on by (most cited first, 30 of 78)

- [Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think](https://mlsanity.com/paper/2410.06940/) (2024)
- [From Slow Bidirectional to Fast Autoregressive Video Diffusion Models](https://mlsanity.com/paper/2412.07772/) (2024)
- [SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models](https://mlsanity.com/paper/2411.05007/) (2024)
- [WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modeling](https://mlsanity.com/paper/2512.14614/) (2025)
- [Diffusion Adversarial Post-Training for One-Step Video Generation](https://mlsanity.com/paper/2501.08316/) (2025)
- [Autoregressive Adversarial Post-Training for Real-Time Interactive Video Generation](https://mlsanity.com/paper/2506.09350/) (2025)
- [SANA-Sprint: One-Step Diffusion with Continuous-Time Consistency Distillation](https://mlsanity.com/paper/2503.09641/) (2025)
- [Phased Consistency Models](https://mlsanity.com/paper/2405.18407/) (2024)
- [What matters for Representation Alignment: Global Information or Spatial Structure?](https://mlsanity.com/paper/2512.10794/) (2025)
- [Learning Few-Step Diffusion Models by Trajectory Distribution Matching](https://mlsanity.com/paper/2503.06674/) (2025)
- [Adversarial Diffusion Compression for Real-World Image Super-Resolution](https://mlsanity.com/paper/2411.13383/) (2024)
- [SeedVR2: One-Step Video Restoration via Diffusion Adversarial Post-Training](https://mlsanity.com/paper/2506.05301/) (2025)
- [FLOAT: Generative Motion Latent Flow Matching for Audio-driven Talking Portrait](https://mlsanity.com/paper/2412.01064/) (2024)
- [InstantDrag: Improving Interactivity in Drag-based Image Editing](https://mlsanity.com/paper/2409.08857/) (2024)
- [LVCD: Reference-based Lineart Video Colorization with Diffusion Models](https://mlsanity.com/paper/2409.12960/) (2024)
- [One-Step Diffusion-Based Image Compression with Semantic Distillation](https://mlsanity.com/paper/2505.16687/) (2025)
- [Layer- and Timestep-Adaptive Differentiable Token Compression Ratios for Efficient Diffusion Transformers](https://mlsanity.com/paper/2412.16822/) (2024)
- [Dual-Expert Consistency Model for Efficient and High-Quality Video Generation](https://mlsanity.com/paper/2506.03123/) (2025)
- [Boosting Latent Diffusion with Perceptual Objectives](https://mlsanity.com/paper/2411.04873/) (2024)
- [CMT: Mid-Training for Efficient Learning of Consistency, Mean Flow, and Flow Map Models](https://mlsanity.com/paper/2509.24526/) (2025)
- [Diffusion Model for Planning: A Systematic Literature Review](https://mlsanity.com/paper/2408.10266/) (2024)
- [Exploring Multimodal Diffusion Transformers for Enhanced Prompt-based Image Editing](https://mlsanity.com/paper/2508.07519/) (2025)
- [Presto! Distilling Steps and Layers for Accelerating Music Generation](https://mlsanity.com/paper/2410.05167/) (2024)
- [Gamma-World: Generative Multi-Agent World Modeling Beyond Two Players](https://mlsanity.com/paper/2605.28816/) (2026)
- [Fast and Memory-Efficient Video Diffusion Using Streamlined Inference](https://mlsanity.com/paper/2411.01171/) (2024)
- [Fast Text-to-Audio Generation with Adversarial Post-Training](https://mlsanity.com/paper/2505.08175/) (2025)
- [PixelGen: Improving Pixel Diffusion with Perceptual Supervision](https://mlsanity.com/paper/2602.02493/) (2026)
- [2.5D Multi-view Averaging Diffusion Model for 3D Medical Image Translation: Application to Low-count PET Reconstruction with CT-less Attenuation Correction](https://mlsanity.com/paper/2406.08374/) (2024)
- [InstantRestore: Single-Step Personalized Face Restoration with Shared-Image Attention](https://mlsanity.com/paper/2412.06753/) (2024)
- [Improved Training Technique for Latent Consistency Models](https://mlsanity.com/paper/2502.01441/) (2025)

## Abstract (arXiv)

We propose a method to distill a complex multistep diffusion model into a single-step conditional GAN student model, dramatically accelerating inference, while preserving image quality. Our approach interprets diffusion distillation as a paired image-to-image translation task, using noise-to-image pairs of the diffusion model's ODE trajectory. For efficient regression loss computation, we propose E-LatentLPIPS, a perceptual loss operating directly in diffusion model's latent space, utilizing an ensemble of augmentations. Furthermore, we adapt a diffusion model to construct a multi-scale discriminator with a text alignment loss to build an effective conditional GAN-based formulation. E-LatentLPIPS converges more efficiently than many existing distillation methods, even accounting for dataset construction costs. We demonstrate that our one-step generator outperforms cutting-edge one-step diffusion distillation models -- DMD, SDXL-Turbo, and SDXL-Lightning -- on the zero-shot COCO benchmark.