# OrthoEraser: Coupled-Neuron Orthogonal Projection for Concept Erasure

arXiv: https://arxiv.org/abs/2603.11493 · HTML: https://mlsanity.com/paper/2603.11493/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Interpreting and steering diffusion models, one of 13 directions in Mechanistic interpretability.
- Builds on 43 of its 50 references that are in the map.
- First posted on arXiv 12 Mar 2026.

## Placed in

- Mechanistic interpretability / [Interpreting and steering diffusion models](https://mlsanity.com/mechinterp/interpreting-and-steering-diffusion-models/)

## Builds on (most cited first, 30 of 43)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation](https://mlsanity.com/paper/2211.12572/) (2022)
- [InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets](https://mlsanity.com/paper/1606.03657/) (2016)
- [Scaling and evaluating sparse autoencoders](https://mlsanity.com/paper/2406.04093/) (2024)
- [Erasing Concepts from Diffusion Models](https://mlsanity.com/paper/2303.07345/) (2023)
- [Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models](https://mlsanity.com/paper/2211.05105/) (2022)
- [Closed-Form Factorization of Latent Semantics in GANs](https://mlsanity.com/paper/2007.06600/) (2020)
- [Unified Concept Editing in Diffusion Models](https://mlsanity.com/paper/2308.14761/) (2023)
- [Diffusion Models already have a Semantic Latent Space](https://mlsanity.com/paper/2210.10960/) (2022)
- [Ablating Concepts in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2303.13516/) (2023)
- [MACE: Mass Concept Erasure in Diffusion Models](https://mlsanity.com/paper/2403.06135/) (2024)
- [Forget-Me-Not: Learning to Forget in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2303.17591/) (2023)
- [Unsupervised Discovery of Interpretable Directions in the GAN Latent Space](https://mlsanity.com/paper/2002.03754/) (2020)
- [Ring-A-Bell! How Reliable are Concept Removal Methods for Diffusion Models?](https://mlsanity.com/paper/2310.10012/) (2023)
- [Show-o2: Improved Native Unified Multimodal Models](https://mlsanity.com/paper/2506.15564/) (2025)
- [SneakyPrompt: Jailbreaking Text-to-image Generative Models](https://mlsanity.com/paper/2305.12082/) (2023)
- [Prompting4Debugging: Red-Teaming Text-to-Image Diffusion Models by Finding Problematic Prompts](https://mlsanity.com/paper/2309.06135/) (2023)
- [One-Dimensional Adapter to Rule Them All: Concepts, Diffusion Models and Erasing Applications](https://mlsanity.com/paper/2312.16145/) (2023)
- [Can Machines Help Us Answering Question 16 in Datasheets, and In Turn Reflecting on Inappropriate Content?](https://mlsanity.com/paper/2202.06675/) (2022)
- [Concept Sliders: LoRA Adaptors for Precise Control in Diffusion Models](https://mlsanity.com/paper/2311.12092/) (2023)
- [Reliable and Efficient Concept Erasure of Text-to-Image Diffusion Models](https://mlsanity.com/paper/2407.12383/) (2024)
- [Receler: Reliable Concept Erasing of Text-to-Image Diffusion Models via Lightweight Erasers](https://mlsanity.com/paper/2311.17717/) (2023)
- [Circumventing Concept Erasure Methods For Text-to-Image Generative Models](https://mlsanity.com/paper/2308.01508/) (2023)
- [SAeUron: Interpretable Concept Unlearning in Diffusion Models with Sparse Autoencoders](https://mlsanity.com/paper/2501.18052/) (2025)
- [Unlearning Concepts in Diffusion Model via Concept Domain Correction and Concept Preserving Gradient](https://mlsanity.com/paper/2405.15304/) (2024)

## Built on by (most cited first, 7 of 7)

- [PEAK: Precise and Persistent Concept Erasure via k-Sparse Autoencoders](https://mlsanity.com/paper/2608.10985/) (2026)
- [No Single Neuron of Failure: Distributed Safety Alignment Against White-Box Attacks](https://mlsanity.com/paper/2608.01414/) (2026)
- [Moving the Safety Barrier: Dynamic Routing Adaptive Alignment Against White-Box Attacks](https://mlsanity.com/paper/2608.02674/) (2026)
- [Video Generation Models: A Survey of Post-Training and Alignment](https://mlsanity.com/paper/2610.00812/) (2026)
- [DV-Lens: Revealing the Functional Organization of Language Model Parameters](https://mlsanity.com/paper/2610.04489/) (2026)
- [Do correct answers have stable latent knowledge support?](https://mlsanity.com/paper/2610.15277/) (2026)
- [SuperAI: Merging Capabilities Across Heterogeneous Foundation Models](https://mlsanity.com/paper/2610.15322/) (2026)

## Abstract (arXiv)

Text-to-image (T2I) models face significant safety risks from adversarial induction, yet current concept erasure methods often cause collateral damage to benign attributes when suppressing selected neurons entirely. This occurs because sensitive and benign semantics exhibit non-orthogonal superposition, sharing activation subspaces where their respective vectors are inherently entangled. To address this issue, we propose OrthoEraser, which leverages sparse autoencoders (SAE) to achieve high-resolution feature disentanglement and subsequently redefines erasure as an analytical orthogonalization projection that preserves the benign manifold's invariance. OrthoEraser first employs SAE to decompose dense activations and segregate sensitive neurons. It then uses coupled neuron detection to identify non-sensitive features vulnerable to intervention. The key novelty lies in an analytical gradient orthogonalization strategy that projects erasure vectors onto the null space of the coupled neurons. This orthogonally decouples the sensitive concepts from the identified critical benign subspace, effectively preserving non-sensitive semantics. Experimental results on safety demonstrate that OrthoEraser achieves high erasure precision, effectively removing harmful content while preserving the integrity of the generative manifold, and significantly outperforming SOTA baselines. This paper contains results of unsafe models.