# SAEs Can Improve Unlearning: Dynamic Sparse Autoencoder Guardrails for Precision Unlearning in LLMs

arXiv: https://arxiv.org/abs/2504.08192 · HTML: https://mlsanity.com/paper/2504.08192/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Placed in Machine unlearning, one of 12 directions in LLM safety and security.
- Builds on 30 of its 43 references that are in the map.
- First posted on arXiv 11 Apr 2025.

## Placed in

- LLM safety and security / [Machine unlearning](https://mlsanity.com/safety/machine-unlearning/)
- Mechanistic interpretability / [SAE features and their applications](https://mlsanity.com/mechinterp/sae-features-and-their-applications/)

## Builds on (most cited first, 30 of 30)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Machine Unlearning](https://mlsanity.com/paper/1912.03817/) (2019)
- [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022)
- [The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning](https://mlsanity.com/paper/2403.03218/) (2024)
- [Knowledge Unlearning for Mitigating Privacy Risks in Language Models](https://mlsanity.com/paper/2210.01504/) (2022)
- [Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models](https://mlsanity.com/paper/2403.19647/) (2024)
- [Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2](https://mlsanity.com/paper/2408.05147/) (2024)
- [Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders](https://mlsanity.com/paper/2407.14435/) (2024)
- [Towards Unbounded Machine Unlearning](https://mlsanity.com/paper/2302.09880/) (2023)
- [MUSE: Machine Unlearning Six-Way Evaluation for Language Models](https://mlsanity.com/paper/2407.06460/) (2024)
- [Rethinking Machine Unlearning for Large Language Models](https://mlsanity.com/paper/2402.08787/) (2024)
- [Fast Machine Unlearning Without Retraining Through Selective Synaptic Dampening](https://mlsanity.com/paper/2308.07707/) (2023)
- [Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning](https://mlsanity.com/paper/2410.07163/) (2024)
- [SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability](https://mlsanity.com/paper/2503.09532/) (2025)
- [An Adversarial Perspective on Machine Unlearning for AI Safety](https://mlsanity.com/paper/2409.18025/) (2024)
- [Steering Language Model Refusal with Sparse Autoencoders](https://mlsanity.com/paper/2411.11296/) (2024)
- [Machine Unlearning: Solutions and Challenges](https://mlsanity.com/paper/2308.07061/) (2023)
- [Do Unlearning Methods Remove Information from Language Model Weights?](https://mlsanity.com/paper/2410.08827/) (2024)
- [Backdoor Defense with Machine Unlearning](https://mlsanity.com/paper/2201.09538/) (2022)
- [Position: LLM Unlearning Benchmarks are Weak Measures of Progress](https://mlsanity.com/paper/2410.02879/) (2024)
- [Applying sparse autoencoders to unlearn knowledge in language models](https://mlsanity.com/paper/2410.19278/) (2024)
- [Obfuscated Activations Bypass LLM Latent-Space Defenses](https://mlsanity.com/paper/2412.09565/) (2024)
- [Unlearning as multi-task optimization: A normalized gradient difference approach with an adaptive learning rate](https://mlsanity.com/paper/2410.22086/) (2024)
- [CaMU: Disentangling Causal Effects in Deep Model Unlearning](https://mlsanity.com/paper/2401.17504/) (2024)

## Built on by (most cited first, 22 of 22)

- [Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Models](https://mlsanity.com/paper/2601.14004/) (2026)
- [Model Unlearning via Sparse Autoencoder Subspace Guided Projections](https://mlsanity.com/paper/2505.24428/) (2025)
- [Position: Mechanistic Interpretability Should Prioritize Feature Consistency in SAEs](https://mlsanity.com/paper/2505.20254/) (2025)
- [Feature Extraction and Steering for Enhanced Chain-of-Thought Reasoning in Language Models](https://mlsanity.com/paper/2505.15634/) (2025)
- [CorrSteer: Generation-Time LLM Steering via Correlated Sparse Autoencoder Features](https://mlsanity.com/paper/2508.12535/) (2025)
- [Precise In-Parameter Concept Erasure in Large Language Models](https://mlsanity.com/paper/2505.22586/) (2025)
- [SoK: Machine Unlearning for Large Language Models](https://mlsanity.com/paper/2506.09227/) (2025)
- [CRISP: Persistent Concept Unlearning via Sparse Autoencoders](https://mlsanity.com/paper/2508.13650/) (2025)
- [Watch the Weights: Unsupervised monitoring and control of fine-tuned LLMs](https://mlsanity.com/paper/2508.00161/) (2025)
- [Signal in the Noise: Polysemantic Interference Transfers and Predicts Cross-Model Influence](https://mlsanity.com/paper/2505.11611/) (2025)
- [NExT-Guard: Training-Free Streaming Safeguard without Token-Level Labels](https://mlsanity.com/paper/2603.02219/) (2026)
- [Navigating by Old Maps: The Pitfalls of Static Mechanistic Localization in LLM Post-Training](https://mlsanity.com/paper/2605.06076/) (2026)
- [Can SAEs reveal and mitigate racial biases of LLMs in healthcare?](https://mlsanity.com/paper/2511.00177/) (2025)
- [Visual Exploration of Feature Relationships in Sparse Autoencoders with Curated Concepts](https://mlsanity.com/paper/2511.06048/) (2025)
- [RippleBench: Capturing Ripple Effects Using Existing Knowledge Repositories](https://mlsanity.com/paper/2512.04144/) (2025)
- [Beyond Forgetting: Representation Misdirection Elicits Controllable Side Behaviors and Capabilities](https://mlsanity.com/paper/2601.21702/) (2026)
- [Detoxifying LLMs via Representation Erasure-Based Preference Optimization](https://mlsanity.com/paper/2602.23391/) (2026)
- [Rotation Control Unlearning: Quantifying and Controlling Continuous Unlearning for LLM with The Cognitive Rotation Space](https://mlsanity.com/paper/2509.25743/) (2025)
- [Test-Time Unlearning via Sparse Autoencoder](https://mlsanity.com/paper/2609.16229/) (2026)
- [Causal Routing for Unlearning](https://mlsanity.com/paper/2609.34475/) (2026)
- [Do LLMs Really Forget? Hidden-State Leakage in Model Unlearning and How to Fix it](https://mlsanity.com/paper/2609.36612/) (2026)
- [Sparse Autoencoders for Mechanistic Interpretability in NLP: A Critical Survey of Recoverability, Interpretability, and Utility](https://mlsanity.com/paper/2610.20139/) (2026)

## Abstract (arXiv)

Machine unlearning is a promising approach to improve LLM safety by removing unwanted knowledge from the model. However, prevailing gradient-based unlearning methods suffer from issues such as high computational costs, hyperparameter instability, poor sequential unlearning capability, vulnerability to relearning attacks, low data efficiency, and lack of interpretability. While Sparse Autoencoders are well-suited to improve these aspects by enabling targeted activation-based unlearning, prior approaches underperform gradient-based methods. This work demonstrates that, contrary to these earlier findings, SAEs can significantly improve unlearning when employed dynamically. We introduce Dynamic DAE Guardrails (DSG), a novel method for precision unlearning that leverages principled feature selection and a dynamic classifier. Our experiments show DSG substantially outperforms leading unlearning methods, achieving superior forget-utility trade-offs. DSG addresses key drawbacks of gradient-based approaches for unlearning -- offering enhanced computational efficiency and stability, robust performance in sequential unlearning, stronger resistance to relearning attacks, better data efficiency including zero-shot settings, and more interpretable unlearning.