# Distilling Diversity and Control in Diffusion Models

arXiv: https://arxiv.org/abs/2503.10637 · HTML: https://mlsanity.com/paper/2503.10637/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 35 of its 44 references that are in the map.
- First posted on arXiv 13 Mar 2025.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 30 of 35)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2308.06721/) (2023)
- [T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.08453/) (2023)
- [Multi-Concept Customization of Text-to-Image Diffusion](https://mlsanity.com/paper/2212.04488/) (2022)
- [One-step Diffusion with Distribution Matching Distillation](https://mlsanity.com/paper/2311.18828/) (2023)
- [Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference](https://mlsanity.com/paper/2310.04378/) (2023)
- [Adversarial Diffusion Distillation](https://mlsanity.com/paper/2311.17042/) (2023)
- [MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing](https://mlsanity.com/paper/2304.08465/) (2023)
- [Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2301.13826/) (2023)
- [Improved Distribution Matching Distillation for Fast Image Synthesis](https://mlsanity.com/paper/2405.14867/) (2024)
- [Erasing Concepts from Diffusion Models](https://mlsanity.com/paper/2303.07345/) (2023)
- [Unified Concept Editing in Diffusion Models](https://mlsanity.com/paper/2308.14761/) (2023)
- [DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data](https://mlsanity.com/paper/2306.09344/) (2023)
- [Ablating Concepts in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2303.13516/) (2023)
- [MACE: Mass Concept Erasure in Diffusion Models](https://mlsanity.com/paper/2403.06135/) (2024)
- [SDXL-Lightning: Progressive Adversarial Diffusion Distillation](https://mlsanity.com/paper/2402.13929/) (2024)
- [Applying Guidance in a Limited Interval Improves Sample and Distribution Quality in Diffusion Models](https://mlsanity.com/paper/2404.07724/) (2024)
- [Break-A-Scene: Extracting Multiple Concepts from a Single Image](https://mlsanity.com/paper/2305.16311/) (2023)
- [Concept Sliders: LoRA Adaptors for Precise Control in Diffusion Models](https://mlsanity.com/paper/2311.12092/) (2023)
- [Concept Decomposition for Visual Exploration and Inspiration](https://mlsanity.com/paper/2305.18203/) (2023)

## Built on by (most cited first, 22 of 22)

- [OneIG-Bench: Omni-dimensional Nuanced Evaluation for Image Generation](https://mlsanity.com/paper/2506.07977/) (2025)
- [When Are Concepts Erased From Diffusion Models?](https://mlsanity.com/paper/2505.17013/) (2025)
- [Fast Text-to-Audio Generation with Adversarial Post-Training](https://mlsanity.com/paper/2505.08175/) (2025)
- [EraseFlow: Learning Concept Erasure Policies via GFlowNet-Driven Alignment](https://mlsanity.com/paper/2511.00804/) (2025)
- [Partition Generative Modeling: Masked Modeling Without Masks](https://mlsanity.com/paper/2505.18883/) (2025)
- [Beyond the Dirac Delta: Mitigating Diversity Collapse in Reinforcement Fine-Tuning for Versatile Image Generation](https://mlsanity.com/paper/2601.12401/) (2026)
- [SubFlow: Sub-mode Conditioned Flow Matching for Diverse One-Step Generation](https://mlsanity.com/paper/2604.12273/) (2026)
- [On-the-fly Repulsion in the Contextual Space for Rich Diversity in Diffusion Transformers](https://mlsanity.com/paper/2603.28762/) (2026)
- [STRIDE: Training-Free Diversity Guidance via PCA-Directed Feature Perturbation in Single-Step Diffusion Models](https://mlsanity.com/paper/2605.11494/) (2026)
- [Uni-DAD: Unified Distillation and Adaptation of Diffusion Models for Few-step Few-shot Image Generation](https://mlsanity.com/paper/2511.18281/) (2025)
- [DefaultShift: Auditing Semantic Default Shift in Accelerated Text-to-Image Models](https://mlsanity.com/paper/2608.21784/) (2026)
- [Manifold-Constrained Noise Optimization for Diverse Diffusion Sampling](https://mlsanity.com/paper/2607.23937/) (2026)
- [Guiding Diffusion with Deep Geometric Moments: Balancing Fidelity and Variation](https://mlsanity.com/paper/2505.12486/) (2025)
- [Diversity Has Always Been There in Your Visual Autoregressive Models](https://mlsanity.com/paper/2511.17074/) (2025)
- [DreamReader: An Interpretability Toolkit for Text-to-Image Models](https://mlsanity.com/paper/2603.13299/) (2026)
- [Restoring Initial Noise Sensitivity in Text-to-Image Distillation via Geometric Alignment](https://mlsanity.com/paper/2606.01651/) (2026)
- [NAMESAKES: Probing Identity Memorization in Text-to-Image Models](https://mlsanity.com/paper/2606.20155/) (2026)
- [Semantic Browsing: Controllable Diversity for Image Generation](https://mlsanity.com/paper/2606.23679/) (2026)
- [Improving Sample Diversity in Autoregressive Text-to-Image Generation via Cluster Truncation](https://mlsanity.com/paper/2607.10535/) (2026)
- [Uncertainty DMD: Restoring Diversity in Few-Step Autoregressive Video Distillation](https://mlsanity.com/paper/2609.11265/) (2026)
- [Accelerating Video Diffusion via Training-Free Trajectory Routing](https://mlsanity.com/paper/2609.30096/) (2026)
- [JIVE: Jacobian-Informed Volume Expansion for Diverse Generative Sampling](https://mlsanity.com/paper/2609.33906/) (2026)

## Abstract (arXiv)

Distilled diffusion models generate images in far fewer timesteps but suffer from reduced sample diversity when generating multiple outputs from the same prompt. To understand this phenomenon, we first investigate whether distillation damages concept representations by examining if the required diversity is properly learned. Surprisingly, distilled models retain the base model's representational structure: control mechanisms like Concept Sliders and LoRAs transfer seamlessly without retraining, and SliderSpace analysis reveals distilled models possess variational directions needed for diversity yet fail to activate them. This redirects our investigation to understanding how the generation dynamics differ between base and distilled models. Using hatx0 trajectory visualization, we discover distilled models commit to their final image structure almost immediately at the first timestep, while base models distribute structural decisions across many steps. To test whether this first-step commitment causes the diversity loss, we introduce diversity distillation, a hybrid approach using the base model for only the first critical timestep before switching to the distilled model. This single intervention restores sample diversity while maintaining computational efficiency. We provide both causal validation and theoretical support showing why the very first timestep concentrates the diversity bottleneck in distilled models. Our code and data are available at https://distillation.baulab.info/