# CASteer: Cross-Attention Steering for Controllable Concept Erasure

arXiv: https://arxiv.org/abs/2503.09630 · HTML: https://mlsanity.com/paper/2503.09630/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in Robustness, privacy and adversarial aspects, one of 14 directions in Diffusion models.
- First posted on arXiv 11 Mar 2025.

## Placed in

- Diffusion models / [Robustness, privacy and adversarial aspects](https://mlsanity.com/diffusion/robustness-privacy-and-adversarial-aspects/)

## Built on by (most cited first, 16 of 16)

- [Set You Straight: Auto-Steering Denoising Trajectories to Sidestep Unwanted Concepts](https://mlsanity.com/paper/2504.12782/) (2025)
- [Erasing Concepts, Steering Generations: A Comprehensive Survey of Concept Suppression](https://mlsanity.com/paper/2505.19398/) (2025)
- [FreeSliders: Training-Free, Modality-Agnostic Concept Sliders for Fine-Grained Diffusion Control in Images, Audio, and Video](https://mlsanity.com/paper/2511.00103/) (2025)
- [TADA! Tuning Audio Diffusion Models through Activation Steering](https://mlsanity.com/paper/2602.11910/) (2026)
- [Whisper Hallucination Detection and Mitigation via Hidden Representation Steering and Sparse AutoEncoders](https://mlsanity.com/paper/2606.07473/) (2026)
- [DTVI: Dual-Stage Textual and Visual Intervention for Safe Text-to-Image Generation](https://mlsanity.com/paper/2603.22041/) (2026)
- [Semantic Steering for Controllable Generation: Tuning-Free Concept Erasure in Multimodal Diffusion Transformers](https://mlsanity.com/paper/2608.12829/) (2026)
- [RASteer: Retain-Aware Activation Steering for Concept Erasure in Diffusion Models](https://mlsanity.com/paper/2610.01969/) (2026)
- [Controllable Coupled Image Generation via Diffusion Models](https://mlsanity.com/paper/2506.06826/) (2025)
- [EDGE-Shield: Efficient Denoising-staGE Shield for Violative Content Filtering via Scalable Reference-Based Matching](https://mlsanity.com/paper/2604.06063/) (2026)
- [Disentangled Anatomy-Disease Diffusion (DADD) for Controllable Ulcerative Colitis Progression Synthesis](https://mlsanity.com/paper/2605.01848/) (2026)
- [BARRIER: Bounded Activation Regions for Robust Information Erasure](https://mlsanity.com/paper/2605.15737/) (2026)
- [Concept Unlearning via Cross-Attention Activation Projection for Diffusion Models](https://mlsanity.com/paper/2605.25765/) (2026)
- [EquiSteer: Cross-Attention Steering Towards a Fairer Text-Guided Image Generation](https://mlsanity.com/paper/2607.01147/) (2026)
- [AcFlow: Controlling Text-to-Image Diffusion Transformers via Learned Conditional Activation Flow](https://mlsanity.com/paper/2609.10723/) (2026)
- [SharedSAE: a unified sparse dictionary across u-net blocks and denoising time for diffusion models](https://mlsanity.com/paper/2610.13997/) (2026)

## Abstract (arXiv)

Diffusion models have transformed image generation, yet controlling their outputs to reliably erase undesired concepts remains challenging. Existing approaches usually require task-specific training and struggle to generalize across both concrete (e.g., objects) and abstract (e.g., styles) concepts. We propose CASteer (Cross-Attention Steering), a training-free framework for concept erasure in diffusion models using steering vectors to influence hidden representations dynamically. CASteer precomputes concept-specific steering vectors by averaging neural activations from images generated for each target concept. During inference, it dynamically applies these vectors to suppress undesired concepts only when they appear, ensuring that unrelated regions remain unaffected. This selective activation enables precise, context-aware erasure without degrading overall image quality. This approach achieves effective removal of harmful or unwanted content across a wide range of visual concepts, all without model retraining. CASteer outperforms state-of-the-art concept erasure techniques while preserving unrelated content and minimizing unintended effects.