# Fine-Grained Erasure in Text-to-Image Diffusion-based Foundation Models

arXiv: https://arxiv.org/abs/2503.19783 · HTML: https://mlsanity.com/paper/2503.19783/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Placed in Robustness, privacy and adversarial aspects, one of 14 directions in Diffusion models.
- Builds on 22 of its 33 references that are in the map.
- First posted on arXiv 25 Mar 2025.

## Placed in

- Diffusion models / [Robustness, privacy and adversarial aspects](https://mlsanity.com/diffusion/robustness-privacy-and-adversarial-aspects/)
- LLM safety and security / [Vision-language and text-to-image safety](https://mlsanity.com/safety/vision-language-and-text-to-image-safety/)

## Builds on (most cited first, 22 of 22)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Erasing Concepts from Diffusion Models](https://mlsanity.com/paper/2303.07345/) (2023)
- [Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models](https://mlsanity.com/paper/2211.05105/) (2022)
- [Unified Concept Editing in Diffusion Models](https://mlsanity.com/paper/2308.14761/) (2023)
- [Are we done with ImageNet?](https://mlsanity.com/paper/2006.07159/) (2020)
- [Ablating Concepts in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2303.13516/) (2023)
- [Forget-Me-Not: Learning to Forget in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2303.17591/) (2023)
- [Ring-A-Bell! How Reliable are Concept Removal Methods for Diffusion Models?](https://mlsanity.com/paper/2310.10012/) (2023)
- [fastai: A Layered API for Deep Learning](https://mlsanity.com/paper/2002.04688/) (2020)
- [One-Dimensional Adapter to Rule Them All: Concepts, Diffusion Models and Erasing Applications](https://mlsanity.com/paper/2312.16145/) (2023)
- [Editable Neural Networks](https://mlsanity.com/paper/2004.00345/) (2020)
- [Receler: Reliable Concept Erasing of Text-to-Image Diffusion Models via Lightweight Erasers](https://mlsanity.com/paper/2311.17717/) (2023)
- [When does dough become a bagel? Analyzing the remaining mistakes on ImageNet](https://mlsanity.com/paper/2205.04596/) (2022)
- [Continual Unlearning for Foundational Text-to-Image Models without Generalization Erosion](https://mlsanity.com/paper/2503.13769/) (2025)
- [Adapt then Unlearn: Exploring Parameter Space Semantics for Unlearning in Generative Adversarial Networks](https://mlsanity.com/paper/2309.14054/) (2023)
- [Automated Classification of Model Errors on ImageNet](https://mlsanity.com/paper/2401.02430/) (2023)

## Built on by (most cited first, 25 of 25)

- [Set You Straight: Auto-Steering Denoising Trajectories to Sidestep Unwanted Concepts](https://mlsanity.com/paper/2504.12782/) (2025)
- [Erasing Concepts, Steering Generations: A Comprehensive Survey of Concept Suppression](https://mlsanity.com/paper/2505.19398/) (2025)
- [SAEmnesia: Erasing Concepts in Diffusion Models with Supervised Sparse Autoencoders](https://mlsanity.com/paper/2509.21379/) (2025)
- [OrthoEraser: Coupled-Neuron Orthogonal Projection for Concept Erasure](https://mlsanity.com/paper/2603.11493/) (2026)
- [Neighbor-Aware Localized Concept Erasure in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2603.25994/) (2026)
- [ACE: Attentional Concept Erasure in Diffusion Models](https://mlsanity.com/paper/2504.11850/) (2025)
- [To Erase, or Not to Erase: Robust Training-Free Concept Erasure with Preservation aware Adaptive Ranked Subspace Expansion](https://mlsanity.com/paper/2607.23492/) (2026)
- [Quantum-Inspired Audio Unlearning: Towards Privacy-Preserving Voice Biometrics](https://mlsanity.com/paper/2507.22208/) (2025)
- [A Unified Framework for Diffusion Model Unlearning with f-Divergence](https://mlsanity.com/paper/2509.21167/) (2025)
- [GrOCE:Graph-Guided Online Concept Erasure for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2511.12968/) (2025)
- [M-ErasureBench: A Comprehensive Multimodal Evaluation Benchmark for Concept Erasure in Diffusion Models](https://mlsanity.com/paper/2512.22877/) (2025)
- [FlowFixer: Towards Detail-Preserving Subject-Driven Generation](https://mlsanity.com/paper/2602.21402/) (2026)
- [Unlearning the Unpromptable: Prompt-free Instance Unlearning in Diffusion Models](https://mlsanity.com/paper/2603.10445/) (2026)
- [Empty SPACE: Cross-Attention Sparsity for Concept Erasure in Diffusion Models](https://mlsanity.com/paper/2605.10198/) (2026)
- [FADE: Frame-Aware Diffusion-Transformer-based Multi-Concept Erasure for Video Unlearning](https://mlsanity.com/paper/2610.03980/) (2026)
- [No Concept Escapes the Audit: Auditing-Aware Unlearning for Verifiable Concept Erasure in Diffusion Models](https://mlsanity.com/paper/2610.05401/) (2026)
- [Your Unlearning Gives You Away: Identifying Erased Concepts in Diffusion Models](https://mlsanity.com/paper/2610.05601/) (2026)
- [Relationship-Aware Safety Unlearning for Multimodal LLMs](https://mlsanity.com/paper/2603.14185/) (2026)
- [Graph Propagated Projection Unlearning: A Unified Framework for Vision and Audio Discriminative Models](https://mlsanity.com/paper/2604.13127/) (2026)
- [BARRIER: Bounded Activation Regions for Robust Information Erasure](https://mlsanity.com/paper/2605.15737/) (2026)
- [Forgetting Has Neighbors: Localized Collateral Forgetting in Machine Unlearning](https://mlsanity.com/paper/2605.31317/) (2026)
- [AutoAnchor: Stable Diffusion Unlearning Using Cross-Attention as a Manifold Surrogate](https://mlsanity.com/paper/2607.08337/) (2026)
- [Stress Testing Concept Erasure with Large Language Model Agents](https://mlsanity.com/paper/2607.17890/) (2026)
- [MapRoute++: Surrogate-Guided Semantic Routing for Visual Concept Unlearning](https://mlsanity.com/paper/2608.13478/) (2026)
- [TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models](https://mlsanity.com/paper/2608.15341/) (2026)

## Abstract (arXiv)

Existing unlearning algorithms in text-to-image generative models often fail to preserve the knowledge of semantically related concepts when removing specific target concepts: a challenge known as adjacency. To address this, we propose FADE (Fine grained Attenuation for Diffusion Erasure), introducing adjacency aware unlearning in diffusion models. FADE comprises two components: (1) the Concept Neighborhood, which identifies an adjacency set of related concepts, and (2) Mesh Modules, employing a structured combination of Expungement, Adjacency, and Guidance loss components. These enable precise erasure of target concepts while preserving fidelity across related and unrelated concepts. Evaluated on datasets like Stanford Dogs, Oxford Flowers, CUB, I2P, Imagenette, and ImageNet1k, FADE effectively removes target concepts with minimal impact on correlated concepts, achieving atleast a 12% improvement in retention performance over state-of-the-art methods.