# Can sparse autoencoders be used to decompose and interpret steering vectors?

arXiv: https://arxiv.org/abs/2411.08790 · HTML: https://mlsanity.com/paper/2411.08790/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Placed in SAE features and their applications, one of 13 directions in Mechanistic interpretability.
- Builds on 17 of its 23 references that are in the map.
- First posted on arXiv 13 Nov 2024.

## Placed in

- Mechanistic interpretability / [SAE features and their applications](https://mlsanity.com/mechinterp/sae-features-and-their-applications/)

## Builds on (most cited first, 17 of 17)

- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Representation Engineering: A Top-Down Approach to AI Transparency](https://mlsanity.com/paper/2310.01405/) (2023)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [Refusal in Language Models Is Mediated by a Single Direction](https://mlsanity.com/paper/2406.11717/) (2024)
- [Steering Language Models With Activation Engineering](https://mlsanity.com/paper/2308.10248/) (2023)
- [Discovering Language Model Behaviors with Model-Written Evaluations](https://mlsanity.com/paper/2212.09251/) (2022)
- [Scaling and evaluating sparse autoencoders](https://mlsanity.com/paper/2406.04093/) (2024)
- [Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models](https://mlsanity.com/paper/2403.19647/) (2024)
- [Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2](https://mlsanity.com/paper/2408.05147/) (2024)
- [Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders](https://mlsanity.com/paper/2407.14435/) (2024)
- [In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steering](https://mlsanity.com/paper/2311.06668/) (2023)
- [Foundational Challenges in Assuring Alignment and Safety of Large Language Models](https://mlsanity.com/paper/2404.09932/) (2024)
- [Analyzing the Generalization and Reliability of Steering Vectors](https://mlsanity.com/paper/2407.12404/) (2024)
- [Extending Activation Steering to Broad Skills and Multiple Behaviours](https://mlsanity.com/paper/2403.05767/) (2024)
- [Model Editing as a Robust and Denoised variant of DPO: A Case Study on Toxicity](https://mlsanity.com/paper/2405.13967/) (2024)
- [Sparse Autoencoders Enable Scalable and Reliable Circuit Identification in Language Models](https://mlsanity.com/paper/2405.12522/) (2024)

## Built on by (most cited first, 19 of 19)

- [AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders](https://mlsanity.com/paper/2501.17148/) (2025)
- [Archetypal SAE: Adaptive and Stable Dictionary Learning for Concept Extraction in Large Vision Models](https://mlsanity.com/paper/2502.12892/) (2025)
- [To Steer or Not to Steer? Mechanistic Error Reduction with Abstention for Language Models](https://mlsanity.com/paper/2510.13290/) (2025)
- [Representation Engineering for Large-Language Models: Survey and Research Challenges](https://mlsanity.com/paper/2502.17601/) (2025)
- [Unveiling Language-Specific Features in Large Language Models via Sparse Autoencoders](https://mlsanity.com/paper/2505.05111/) (2025)
- [Steering Vector Fields for Context-Aware Inference-Time Control in Large Language Models](https://mlsanity.com/paper/2602.01654/) (2026)
- [Does higher interpretability imply better utility? A Pairwise Analysis on Sparse Autoencoders](https://mlsanity.com/paper/2510.03659/) (2025)
- [From Weights to Activations: Is Steering the Next Frontier of Adaptation?](https://mlsanity.com/paper/2604.14090/) (2026)
- [EasyEdit2: An Easy-to-use Steering Framework for Editing Large Language Models](https://mlsanity.com/paper/2504.15133/) (2025)
- [DLM-Scope: Mechanistic Interpretability of Diffusion Language Models via Sparse Autoencoders](https://mlsanity.com/paper/2602.05859/) (2026)
- [Evaluate Bias without Manual Test Sets: A Concept Representation Perspective for LLMs](https://mlsanity.com/paper/2505.15524/) (2025)
- [A Comparative Analysis of Sparse Autoencoder and Activation Difference in Language Model Steering](https://mlsanity.com/paper/2510.01246/) (2025)
- [SASFT: Sparse Autoencoder-guided Supervised Finetuning to Mitigate Unexpected Code-Switching in LLMs](https://mlsanity.com/paper/2507.14894/) (2025)
- [Rethinking Sparse Autoencoders: Select-and-Project for Fairness and Control from Encoder Features Alone](https://mlsanity.com/paper/2509.10809/) (2025)
- [Polarity-Aware Probing for Quantifying Latent Alignment in Language Models](https://mlsanity.com/paper/2511.21737/) (2025)
- [Auditing Asset-Specific Preferences in Financial Large Language Models: Evidence from Bitcoin Representations and Portfolio Allocation](https://mlsanity.com/paper/2606.02528/) (2026)
- [Look But Don't Touch with Sparse Autoencoders for Unlearning in Diffusion Models](https://mlsanity.com/paper/2606.31699/) (2026)
- [Beyond a Bag of Features: Set-Level Instability in Sparse Autoencoders](https://mlsanity.com/paper/2608.11197/) (2026)
- [Steering Fields: Adaptive Vector Fields for Safe Image Generation and Beyond](https://mlsanity.com/paper/2609.39573/) (2026)

## Abstract (arXiv)

Steering vectors are a promising approach to control the behaviour of large language models. However, their underlying mechanisms remain poorly understood. While sparse autoencoders (SAEs) may offer a potential method to interpret steering vectors, recent findings show that SAE-reconstructed vectors often lack the steering properties of the original vectors. This paper investigates why directly applying SAEs to steering vectors yields misleading decompositions, identifying two reasons: (1) steering vectors fall outside the input distribution for which SAEs are designed, and (2) steering vectors can have meaningful negative projections in feature directions, which SAEs are not designed to accommodate. These limitations hinder the direct use of SAEs for interpreting steering vectors.