# Improving Steering Vectors by Targeting Sparse Autoencoder Features

arXiv: https://arxiv.org/abs/2411.02193 · HTML: https://mlsanity.com/paper/2411.02193/

## Key facts

- Cited by 92 papers in the map (data through 14 Oct 2026).
- Placed in SAE features and their applications, one of 13 directions in Mechanistic interpretability.
- Builds on 14 of its 40 references that are in the map.
- First posted on arXiv 4 Nov 2024.

## Placed in

- Mechanistic interpretability / [SAE features and their applications](https://mlsanity.com/mechinterp/sae-features-and-their-applications/)

## Builds on (most cited first, 14 of 14)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Jailbroken: How Does LLM Safety Training Fail?](https://mlsanity.com/paper/2307.02483/) (2023)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Representation Engineering: A Top-Down Approach to AI Transparency](https://mlsanity.com/paper/2310.01405/) (2023)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022)
- [The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions](https://mlsanity.com/paper/2404.13208/) (2024)
- [Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models](https://mlsanity.com/paper/2403.19647/) (2024)
- [Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2](https://mlsanity.com/paper/2408.05147/) (2024)
- [Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders](https://mlsanity.com/paper/2407.14435/) (2024)
- [Programming Refusal with Conditional Activation Steering](https://mlsanity.com/paper/2409.05907/) (2024)
- [Linear Algebraic Structure of Word Senses, with Applications to Polysemy](https://mlsanity.com/paper/1601.03764/) (2016)
- [Personalized Steering of Large Language Models: Versatile Steering Vectors Through Bi-directional Preference Optimization](https://mlsanity.com/paper/2406.00045/) (2024)
- [Steering Knowledge Selection Behaviours in LLMs via SAE-Based Representation Engineering](https://mlsanity.com/paper/2410.15999/) (2024)

## Built on by (most cited first, 30 of 92)

- [AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders](https://mlsanity.com/paper/2501.17148/) (2025)
- [Are Sparse Autoencoders Useful? A Case Study in Sparse Probing](https://mlsanity.com/paper/2502.16681/) (2025)
- [Steering Language Model Refusal with Sparse Autoencoders](https://mlsanity.com/paper/2411.11296/) (2024)
- [Persona Features Control Emergent Misalignment](https://mlsanity.com/paper/2506.19823/) (2025)
- [A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Models](https://mlsanity.com/paper/2503.05613/) (2025)
- [Enhancing Automated Interpretability with Output-Centric Feature Descriptions](https://mlsanity.com/paper/2501.08319/) (2025)
- [Interpretable Steering of Large Language Models with Feature Guided Activation Additions](https://mlsanity.com/paper/2501.09929/) (2025)
- [SAE-SSV: Supervised Steering in Sparse Representation Spaces for Reliable Control of Language Models](https://mlsanity.com/paper/2505.16188/) (2025)
- [Beyond Prompt Engineering: Robust Behavior Control in LLMs via Steering Target Atoms](https://mlsanity.com/paper/2505.20322/) (2025)
- [Representation Engineering for Large-Language Models: Survey and Research Challenges](https://mlsanity.com/paper/2502.17601/) (2025)
- [Tracking the Feature Dynamics in LLM Training: A Mechanistic Study](https://mlsanity.com/paper/2412.17626/) (2024)
- [Does Time Have Its Place? Temporal Heads: Where Language Models Recall Time-specific Information](https://mlsanity.com/paper/2502.14258/) (2025)
- [Position: Mechanistic Interpretability Should Prioritize Feature Consistency in SAEs](https://mlsanity.com/paper/2505.20254/) (2025)
- [Analyze Feature Flow to Enhance Interpretation and Steering in Language Models](https://mlsanity.com/paper/2502.03032/) (2025)
- [SAUCE: Selective Concept Unlearning in Vision-Language Models with Sparse Autoencoders](https://mlsanity.com/paper/2503.14530/) (2025)
- [Evaluating Adversarial Robustness of Concept Representations in Sparse Autoencoders](https://mlsanity.com/paper/2505.16004/) (2025)
- [Causal Language Control in Multilingual Transformers via Sparse Feature Steering](https://mlsanity.com/paper/2507.13410/) (2025)
- [CorrSteer: Generation-Time LLM Steering via Correlated Sparse Autoencoder Features](https://mlsanity.com/paper/2508.12535/) (2025)
- [Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safety](https://mlsanity.com/paper/2506.05451/) (2025)
- [Does higher interpretability imply better utility? A Pairwise Analysis on Sparse Autoencoders](https://mlsanity.com/paper/2510.03659/) (2025)
- [Analysing the Safety Pitfalls of Steering Vectors](https://mlsanity.com/paper/2603.24543/) (2026)
- [Mitigating Memorization in LLMs using Activation Steering](https://mlsanity.com/paper/2503.06040/) (2025)
- [EasyEdit2: An Easy-to-use Steering Framework for Editing Large Language Models](https://mlsanity.com/paper/2504.15133/) (2025)
- [Interpreting Large Text-to-Image Diffusion Models with Dictionary Learning](https://mlsanity.com/paper/2505.24360/) (2025)
- [Line of Sight: On Linear Representations in VLLMs](https://mlsanity.com/paper/2506.04706/) (2025)
- [Steering Safely or Off a Cliff? Rethinking Specificity and Robustness in Inference-Time Interventions](https://mlsanity.com/paper/2602.06256/) (2026)
- [The Geometry of Representational Failures in Vision Language Models](https://mlsanity.com/paper/2602.07025/) (2026)
- [Stop Probing, Start Coding: Why Linear Probes and Sparse Autoencoders Fail at Compositional Generalisation](https://mlsanity.com/paper/2603.28744/) (2026)
- [Neurons Speak in Ranges: Breaking Free from Discrete Neuronal Attribution](https://mlsanity.com/paper/2502.06809/) (2025)
- [The Anatomy of Alignment: Decomposing Preference Optimization by Steering Sparse Features](https://mlsanity.com/paper/2509.12934/) (2025)

## Abstract (arXiv)

To control the behavior of language models, steering methods attempt to ensure that outputs of the model satisfy specific pre-defined properties. Adding steering vectors to the model is a promising method of model control that is easier than finetuning, and may be more robust than prompting. However, it can be difficult to anticipate the effects of steering vectors produced by methods such as CAA [Panickssery et al., 2024] or the direct use of SAE latents [Templeton et al., 2024]. In our work, we address this issue by using SAEs to measure the effects of steering vectors, giving us a method that can be used to understand the causal effect of any steering vector intervention. We use this method for measuring causal effects to develop an improved steering method, SAE-Targeted Steering (SAE-TS), which finds steering vectors to target specific SAE features while minimizing unintended side effects. We show that overall, SAE-TS balances steering effects with coherence better than CAA and SAE feature steering, when evaluated on a range of tasks.