# UnGuide: Learning to Forget with LoRA-Guided Diffusion Models

arXiv: https://arxiv.org/abs/2508.05755 · HTML: https://mlsanity.com/paper/2508.05755/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Robustness, privacy and adversarial aspects, one of 14 directions in Diffusion models.
- Builds on 15 of its 26 references that are in the map.
- First posted on arXiv 7 Aug 2025.

## Placed in

- Diffusion models / [Robustness, privacy and adversarial aspects](https://mlsanity.com/diffusion/robustness-privacy-and-adversarial-aspects/)
- LLM safety and security / [Vision-language and text-to-image safety](https://mlsanity.com/safety/vision-language-and-text-to-image-safety/)

## Builds on (most cited first, 15 of 15)

- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Muse: Text-To-Image Generation via Masked Generative Transformers](https://mlsanity.com/paper/2301.00704/) (2023)
- [Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models](https://mlsanity.com/paper/2211.05105/) (2022)
- [SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generation](https://mlsanity.com/paper/2310.12508/) (2023)
- [CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformers](https://mlsanity.com/paper/2204.14217/) (2022)
- [MACE: Mass Concept Erasure in Diffusion Models](https://mlsanity.com/paper/2403.06135/) (2024)
- [Guiding a Diffusion Model with a Bad Version of Itself](https://mlsanity.com/paper/2406.02507/) (2024)
- [Red-Teaming the Stable Diffusion Safety Filter](https://mlsanity.com/paper/2210.04610/) (2022)
- [TF-ICON: Diffusion-Based Training-Free Cross-Domain Image Composition](https://mlsanity.com/paper/2307.12493/) (2023)
- [One-Dimensional Adapter to Rule Them All: Concepts, Diffusion Models and Erasing Applications](https://mlsanity.com/paper/2312.16145/) (2023)
- [Scissorhands: Scrub Data Influence via Connection Sensitivity in Networks](https://mlsanity.com/paper/2401.06187/) (2024)
- [Classifier-free Guidance with Adaptive Scaling](https://mlsanity.com/paper/2502.10574/) (2025)
- [SEMU: Singular Value Decomposition for Efficient Machine Unlearning](https://mlsanity.com/paper/2502.07587/) (2025)
- [GeoGuide: Geometric guidance of diffusion models](https://mlsanity.com/paper/2407.12889/) (2024)
- [AutoLoRA: AutoGuidance Meets Low-Rank Adaptation for Diffusion Models](https://mlsanity.com/paper/2410.03941/) (2024)

## Built on by (most cited first, 7 of 7)

- [Multimodal Unlearning Across Vision, Language, Video, and Audio: Survey of Methods, Datasets, and Benchmarks](https://mlsanity.com/paper/2607.07907/) (2026)
- [Locality-Aware Continual Unlearning for Diffusion Models](https://mlsanity.com/paper/2512.02657/) (2025)
- [UnHype: CLIP-Guided Hypernetworks for Dynamic LoRA Unlearning](https://mlsanity.com/paper/2602.03410/) (2026)
- [Memory Self-Regeneration: Uncovering Hidden Knowledge in Unlearned Models](https://mlsanity.com/paper/2510.03263/) (2025)
- [From Unlearning to UNBRANDING: A Benchmark for Trademark-Safe Text-to-Image Generation](https://mlsanity.com/paper/2512.13953/) (2025)
- [BARRIER: Bounded Activation Regions for Robust Information Erasure](https://mlsanity.com/paper/2605.15737/) (2026)
- [FOMO: Forget the Concept, Don't Miss Out on the Scene in Selective Video Unlearning](https://mlsanity.com/paper/2609.39605/) (2026)

## Abstract (arXiv)

Recent advances in large-scale text-to-image diffusion models have heightened concerns about their potential misuse, especially in generating harmful or misleading content. This underscores the urgent need for effective machine unlearning, i.e., removing specific knowledge or concepts from pretrained models without compromising overall performance. One possible approach is Low-Rank Adaptation (LoRA), which offers an efficient means to fine-tune models for targeted unlearning. However, LoRA often inadvertently alters unrelated content, leading to diminished image fidelity and realism. To address this limitation, we introduce UnGuide -- a novel approach which incorporates UnGuidance, a dynamic inference mechanism that leverages Classifier-Free Guidance (CFG) to exert precise control over the unlearning process. UnGuide modulates the guidance scale based on the stability of a few first steps of denoising processes, enabling selective unlearning by LoRA adapter. For prompts containing the erased concept, the LoRA module predominates and is counterbalanced by the base model; for unrelated prompts, the base model governs generation, preserving content fidelity. Empirical results demonstrate that UnGuide achieves controlled concept removal and retains the expressive power of diffusion models, outperforming existing LoRA-based methods in both object erasure and explicit content removal tasks.