# Adversarial Score identity Distillation: Rapidly Surpassing the Teacher in One Step

arXiv: https://arxiv.org/abs/2410.14919 · HTML: https://mlsanity.com/paper/2410.14919/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 82 of its 107 references that are in the map.
- First posted on arXiv 19 Oct 2024.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)
- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 30 of 82)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps](https://mlsanity.com/paper/2206.00927/) (2022)
- [Consistency Models](https://mlsanity.com/paper/2303.01469/) (2023)
- [Normalizing Flows for Probabilistic Modeling and Inference](https://mlsanity.com/paper/1912.02762/) (2019)
- [ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation](https://mlsanity.com/paper/2305.16213/) (2023)
- [Variational Diffusion Models](https://mlsanity.com/paper/2107.00630/) (2021)
- [MaskGIT: Masked Generative Image Transformer](https://mlsanity.com/paper/2202.04200/) (2022)
- [StarGAN v2: Diverse Image Synthesis for Multiple Domains](https://mlsanity.com/paper/1912.01865/) (2019)
- [Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction](https://mlsanity.com/paper/2404.02905/) (2024)
- [One-step Diffusion with Distribution Matching Distillation](https://mlsanity.com/paper/2311.18828/) (2023)

## Built on by (most cited first, 30 of 47)

- [Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion](https://mlsanity.com/paper/2506.08009/) (2025)
- [Align Your Flow: Scaling Continuous-Time Flow Map Distillation](https://mlsanity.com/paper/2506.14603/) (2025)
- [End-to-End Training for Autoregressive Video Diffusion via Self-Resampling](https://mlsanity.com/paper/2512.15702/) (2025)
- [Decoupled DMD: CFG Augmentation as the Spear, Distribution Matching as the Shield](https://mlsanity.com/paper/2511.22677/) (2025)
- [Adversarial Distribution Matching for Diffusion Distillation Towards Efficient Image and Video Synthesis](https://mlsanity.com/paper/2507.18569/) (2025)
- [Direct Discriminative Optimization: Your Likelihood-Based Visual Generative Model is Secretly a GAN Discriminator](https://mlsanity.com/paper/2503.01103/) (2025)
- [DLM-One: Diffusion Language Models for One-Step Sequence Generation](https://mlsanity.com/paper/2506.00290/) (2025)
- [Diversity-Preserved Distribution Matching Distillation for Fast Visual Synthesis](https://mlsanity.com/paper/2602.03139/) (2026)
- [Visual Generation Without Guidance](https://mlsanity.com/paper/2501.15420/) (2025)
- [Ultra-Fast Language Generation via Discrete Diffusion Divergence Instruct](https://mlsanity.com/paper/2509.25035/) (2025)
- [Flow Map Distillation Without Data](https://mlsanity.com/paper/2511.19428/) (2025)
- [Uni-Instruct: One-step Diffusion Model through Unified Diffusion Divergence Instruction](https://mlsanity.com/paper/2505.20755/) (2025)
- [Ideas in Inference-time Scaling can Benefit Generative Pre-training Algorithms](https://mlsanity.com/paper/2503.07154/) (2025)
- [Few-Step Diffusion via Score identity Distillation](https://mlsanity.com/paper/2505.12674/) (2025)
- [Accelerating Video Diffusion Models via Distribution Matching](https://mlsanity.com/paper/2412.05899/) (2024)
- [Learning to Integrate Diffusion ODEs by Averaging the Derivatives](https://mlsanity.com/paper/2505.14502/) (2025)
- [Parallel Decoding Distillation for Fast Image and Video Generation](https://mlsanity.com/paper/2607.26004/) (2026)
- [DiffRatio: Training One-Step Diffusion Models Without Teacher Supervision](https://mlsanity.com/paper/2502.08005/) (2025)
- [Score Distillation of Flow Matching Models](https://mlsanity.com/paper/2509.25127/) (2025)
- [IDLM: Inverse-distilled Diffusion Language Models](https://mlsanity.com/paper/2602.19066/) (2026)
- [SCoT: Unifying Consistency Models and Rectified Flows via Straight-Consistent Trajectories](https://mlsanity.com/paper/2502.16972/) (2025)
- [RAVEN: Real-time Autoregressive Video Extrapolation with Consistency-model GRPO](https://mlsanity.com/paper/2605.15190/) (2026)
- [Soft-Di[M]O: Improving One-Step Discrete Image Generation with Soft Embeddings](https://mlsanity.com/paper/2509.22925/) (2025)
- [A Survey on Pre-Trained Diffusion Model Distillations](https://mlsanity.com/paper/2502.08364/) (2025)
- [Dimathtt[M]O: Distilling Masked Diffusion Models into One-step Generator](https://mlsanity.com/paper/2503.15457/) (2025)
- [Universal Inverse Distillation for Matching Models with Real-Data Supervision (No GANs)](https://mlsanity.com/paper/2509.22459/) (2025)
- [Neon: Negative Extrapolation From Self-Training Improves Image Generation](https://mlsanity.com/paper/2510.03597/) (2025)
- [FSVideo: Fast Speed Video Diffusion Model in a Highly-Compressed Latent Space](https://mlsanity.com/paper/2602.02092/) (2026)
- [Self-Evaluation Unlocks Any-Step Text-to-Image Generation](https://mlsanity.com/paper/2512.22374/) (2025)
- [Toward Theoretical Insights into Diffusion Trajectory Distillation via Operator Merging](https://mlsanity.com/paper/2505.16024/) (2025)

## Abstract (arXiv)

Score identity Distillation (SiD) is a data-free method that has achieved SOTA performance in image generation by leveraging only a pretrained diffusion model, without requiring any training data. However, its ultimate performance is constrained by how accurate the pretrained model captures the true data scores at different stages of the diffusion process. In this paper, we introduce SiDA (SiD with Adversarial Loss), which not only enhances generation quality but also improves distillation efficiency by incorporating real images and adversarial loss. SiDA utilizes the encoder from the generator's score network as a discriminator, allowing it to distinguish between real images and those generated by SiD. The adversarial loss is batch-normalized within each GPU and then combined with the original SiD loss. This integration effectively incorporates the average "fakeness" per GPU batch into the pixel-based SiD loss, enabling SiDA to distill a single-step generator. SiDA converges significantly faster than its predecessor when distilled from scratch, and swiftly improves upon the original model's performance during fine-tuning from a pre-distilled SiD generator. This one-step adversarial distillation method establishes new benchmarks in generation performance when distilling EDM diffusion models, achieving FID scores of 1.110 on ImageNet 64x64. When distilling EDM2 models trained on ImageNet 512x512, our SiDA method surpasses even the largest teacher model, EDM2-XXL, which achieved an FID of 1.81 using classifier-free guidance (CFG) and 63 generation steps. In contrast, SiDA achieves FID scores of 2.156 for size XS, 1.669 for S, 1.488 for M, 1.413 for L, 1.379 for XL, and 1.366 for XXL, all without CFG and in a single generation step. These results highlight substantial improvements across all model sizes. Our code is available at https://github.com/mingyuanzhou/SiD/tree/sida.