# LoRA-Enhanced Distillation on Guided Diffusion Models

arXiv: https://arxiv.org/abs/2312.06899 · HTML: https://mlsanity.com/paper/2312.06899/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 6 of its 8 references that are in the map.
- First posted on arXiv 12 Dec 2023.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 6 of 6)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [On Distillation of Guided Diffusion Models](https://mlsanity.com/paper/2210.03142/) (2022)
- [DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Models](https://mlsanity.com/paper/2210.14896/) (2022)
- [Selective Guidance: Are All the Denoising Steps of Guided Diffusion Important?](https://mlsanity.com/paper/2305.09847/) (2023)

## Built on by (most cited first, 5 of 5)

- [A Survey on LoRA of Large Language Models](https://mlsanity.com/paper/2407.11046/) (2024)
- [LoRA-Ensemble: Efficient Uncertainty Modelling for Self-Attention Networks](https://mlsanity.com/paper/2405.14438/) (2024)
- [Anomaly Detection and Generation with Diffusion Models: A Survey](https://mlsanity.com/paper/2506.09368/) (2025)
- [Simple Drop-in LoRA Conditioning on Attention Layers Will Improve Your Diffusion Model](https://mlsanity.com/paper/2405.03958/) (2024)
- [The Fabrication of Reality and Fantasy: Scene Generation with LLM-Assisted Prompt Interpretation](https://mlsanity.com/paper/2407.12579/) (2024)

## Abstract (arXiv)

Diffusion models, such as Stable Diffusion (SD), offer the ability to generate high-resolution images with diverse features, but they come at a significant computational and memory cost. In classifier-free guided diffusion models, prolonged inference times are attributed to the necessity of computing two separate diffusion models at each denoising step. Recent work has shown promise in improving inference time through distillation techniques, teaching the model to perform similar denoising steps with reduced computations. However, the application of distillation introduces additional memory overhead to these already resource-intensive diffusion models, making it less practical. To address these challenges, our research explores a novel approach that combines Low-Rank Adaptation (LoRA) with model distillation to efficiently compress diffusion models. This approach not only reduces inference time but also mitigates memory overhead, and notably decreases memory consumption even before applying distillation. The results are remarkable, featuring a significant reduction in inference time due to the distillation process and a substantial 50% reduction in memory consumption. Our examination of the generated images underscores that the incorporation of LoRA-enhanced distillation maintains image quality and alignment with the provided prompts. In summary, while conventional distillation tends to increase memory consumption, LoRA-enhanced distillation offers optimization without any trade-offs or compromises in quality.