# Stable and Steerable Sparse Autoencoders with Weight Regularization

arXiv: https://arxiv.org/abs/2603.04198 · HTML: https://mlsanity.com/paper/2603.04198/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in SAE features and their applications, one of 13 directions in Mechanistic interpretability.
- Builds on 13 of its 18 references that are in the map.
- First posted on arXiv 4 Mar 2026.

## Placed in

- Mechanistic interpretability / [SAE features and their applications](https://mlsanity.com/mechinterp/sae-features-and-their-applications/)

## Builds on (most cited first, 13 of 13)

- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022)
- [Scaling and evaluating sparse autoencoders](https://mlsanity.com/paper/2406.04093/) (2024)
- [Learning Multi-Level Features with Matryoshka Sparse Autoencoders](https://mlsanity.com/paper/2503.17547/) (2025)
- [BatchTopK Sparse Autoencoders](https://mlsanity.com/paper/2412.06410/) (2024)
- [SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability](https://mlsanity.com/paper/2503.09532/) (2025)
- [Are Sparse Autoencoders Useful? A Case Study in Sparse Probing](https://mlsanity.com/paper/2502.16681/) (2025)
- [Sparse Autoencoders Trained on the Same Data Learn Different Features](https://mlsanity.com/paper/2501.16615/) (2025)
- [Identifying Functionally Important Features with End-to-End Sparse Dictionary Learning](https://mlsanity.com/paper/2405.12241/) (2024)
- [Enhancing Neural Network Interpretability with Feature-Aligned Sparse Autoencoders](https://mlsanity.com/paper/2411.01220/) (2024)
- [Interpretability as Compression: Reconsidering SAE Explanations of Neural Activations with MDL-SAEs](https://mlsanity.com/paper/2410.11179/) (2024)
- [Sparse but Wrong: Incorrect L0 Leads to Incorrect Features in Sparse Autoencoders](https://mlsanity.com/paper/2508.16560/) (2025)
- [Attribution-Guided Distillation of Matryoshka Sparse Autoencoders](https://mlsanity.com/paper/2512.24975/) (2025)

## Built on by (most cited first, 5 of 5)

- [Perplexity Can Miss SAE Feature Damage Under Quantization](https://mlsanity.com/paper/2606.03002/) (2026)
- [Probing and steering biology across Boltz-1s trunk-diffusion boundary](https://mlsanity.com/paper/2608.11475/) (2026)
- [Size Doesn't Matter: Cosine-Scored Sparse Autoencoders](https://mlsanity.com/paper/2606.15054/) (2026)
- [When Pruning Meets Interpretability: Preserving Sparse Autoencoder Robustness in LLMs](https://mlsanity.com/paper/2608.25941/) (2026)
- [Efficient Auto-Interpretability of AI Models in Biology](https://mlsanity.com/paper/2608.27754/) (2026)

## Abstract (arXiv)

Sparse autoencoders (SAEs) are widely used to extract human-interpretable features from neural network activations, but their learned features can vary substantially across random seeds and training choices. To improve stability, we studied weight regularization by adding L1 or L2 penalties on encoder and decoder weights, and evaluate how regularization interacts with common SAE training defaults. On MNIST, we observe that L2 weight regularization produces a core of highly aligned features and, when combined with tied initialization and unit-norm decoder constraints, it dramatically increases cross-seed feature consistency. For TopK SAEs trained on language model activations (Pythia-70M-deduped), adding a small L2 weight penalty increased the fraction of features shared across three random seeds and roughly doubles steering success rates, while leaving the mean of automated interpretability scores essentially unchanged. Finally, in the regularized setting, activation steering success becomes better predicted by auto-interpretability scores, suggesting that regularization can align text-based feature explanations with functional controllability.