# RASteer: Retain-Aware Activation Steering for Concept Erasure in Diffusion Models arXiv: https://arxiv.org/abs/2610.01969 ยท HTML: https://mlsanity.com/paper/2610.01969/ ## Key facts - Cited by 1 papers in the map (data through 14 Oct 2026). - Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. - Builds on 46 of its 50 references that are in the map. - First posted on arXiv 1 Oct 2026. ## Placed in - Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/) - Mechanistic interpretability / [Interpreting and steering diffusion models](https://mlsanity.com/mechinterp/interpreting-and-steering-diffusion-models/) ## Builds on (most cited first, 30 of 46) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021) - [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014) - [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021) - [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018) - [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023) - [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022) - [Mass-Editing Memory in a Transformer](https://mlsanity.com/paper/2210.07229/) (2022) - [Erasing Concepts from Diffusion Models](https://mlsanity.com/paper/2303.07345/) (2023) - [Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models](https://mlsanity.com/paper/2211.05105/) (2022) - [Unified Concept Editing in Diffusion Models](https://mlsanity.com/paper/2308.14761/) (2023) - [Ablating Concepts in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2303.13516/) (2023) - [MACE: Mass Concept Erasure in Diffusion Models](https://mlsanity.com/paper/2403.06135/) (2024) - [Forget-Me-Not: Learning to Forget in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2303.17591/) (2023) - [Red-Teaming the Stable Diffusion Safety Filter](https://mlsanity.com/paper/2210.04610/) (2022) - [Mnemonics Training: Multi-Class Incremental Learning without Forgetting](https://mlsanity.com/paper/2002.10211/) (2020) - [One-Dimensional Adapter to Rule Them All: Concepts, Diffusion Models and Erasing Applications](https://mlsanity.com/paper/2312.16145/) (2023) - [Adaptive Aggregation Networks for Class-Incremental Learning](https://mlsanity.com/paper/2010.05063/) (2020) - [Reliable and Efficient Concept Erasure of Text-to-Image Diffusion Models](https://mlsanity.com/paper/2407.12383/) (2024) - [SAFREE: Training-Free and Adaptive Guard for Safe Text-to-Image And Video Generation](https://mlsanity.com/paper/2410.12761/) (2024) - [Editing Implicit Assumptions in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2303.08084/) (2023) - [Receler: Reliable Concept Erasing of Text-to-Image Diffusion Models via Lightweight Erasers](https://mlsanity.com/paper/2311.17717/) (2023) - [Model Editing Harms General Abilities of Large Language Models: Regularization to the Rescue](https://mlsanity.com/paper/2401.04700/) (2024) - [WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models](https://mlsanity.com/paper/2405.14768/) (2024) - [SAeUron: Interpretable Concept Unlearning in Diffusion Models with Sparse Autoencoders](https://mlsanity.com/paper/2501.18052/) (2025) - [RMM: Reinforced Memory Management for Class-Incremental Learning](https://mlsanity.com/paper/2301.05792/) (2023) - [Continual Detection Transformer for Incremental Object Detection](https://mlsanity.com/paper/2304.03110/) (2023) - [Class-Incremental Exemplar Compression for Class-Incremental Learning](https://mlsanity.com/paper/2303.14042/) (2023) - [Continual Learning for Abdominal Multi-Organ and Tumor Segmentation](https://mlsanity.com/paper/2306.00988/) (2023) - [Online Optimal Control with Linear Dynamics and Predictions: Algorithms and Regret Analysis](https://mlsanity.com/paper/1906.11378/) (2019) ## Built on by (most cited first, 1 of 1) - [Beyond Bias Scores: Answerability Confounds in Activation Steering with Gradient-Constrained Direction Design (GCD)](https://mlsanity.com/paper/2610.17641/) (2026) ## Abstract (arXiv) Concept erasure aims to remove a target concept, such as a copyrighted style, a recognizable character, or unsafe content, from a pretrained text-to-image diffusion model while preserving its ability to generate other content. Existing activation steering methods build an erasure direction mainly from the target concept and adjust model activations along it at inference time. However, target and retained concepts often overlap in the model's representation space, so this direction also contains shared components that retained concepts rely on. Steering directly along this direction can therefore suppress retained concepts and harm the generation of non-target content. To address this issue, we propose Retain-aware Activation Steering (RASteer), a training-free method. RASteer first builds a retain subspace from the concepts to preserve. Retain-Orthogonal Steering (ROS) then removes components aligned with this subspace from the erasure direction, making steering more specific to the target. Since fully removing the shared components can weaken erasure, we further introduce Overlap-Adaptive Calibration (OAC). At each layer and denoising step, OAC uses the overlap between the erasure direction and the retain subspace to control how much of each shared component is removed, balancing target erasure and concept preservation. Experiments on unsafe-content, instance, and artistic-style erasure across multiple backbones and benchmarks show that RASteer matches or outperforms the activation steering and weight editing baselines we evaluate, achieving a better balance between erasure and preservation.