# Enhancing LLM Steering through Sparse Autoencoder-Based Vector Refinement

arXiv: https://arxiv.org/abs/2509.23799 · HTML: https://mlsanity.com/paper/2509.23799/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in SAE features and their applications, one of 13 directions in Mechanistic interpretability.
- Builds on 27 of its 34 references that are in the map.
- First posted on arXiv 28 Sep 2025.

## Placed in

- Mechanistic interpretability / [SAE features and their applications](https://mlsanity.com/mechinterp/sae-features-and-their-applications/)

## Builds on (most cited first, 27 of 27)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [Representation Engineering: A Top-Down Approach to AI Transparency](https://mlsanity.com/paper/2310.01405/) (2023)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [Refusal in Language Models Is Mediated by a Single Direction](https://mlsanity.com/paper/2406.11717/) (2024)
- [Discovering Language Model Behaviors with Model-Written Evaluations](https://mlsanity.com/paper/2212.09251/) (2022)
- [AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders](https://mlsanity.com/paper/2501.17148/) (2025)
- [Open Problems in Mechanistic Interpretability](https://mlsanity.com/paper/2501.16496/) (2025)
- [Analyzing the Generalization and Reliability of Steering Vectors](https://mlsanity.com/paper/2407.12404/) (2024)
- [Do I Know This Entity? Knowledge Awareness and Hallucinations in Language Models](https://mlsanity.com/paper/2411.14257/) (2024)
- [SAEs Are Good for Steering -- If You Select the Right Features](https://mlsanity.com/paper/2505.20063/) (2025)
- [A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Models](https://mlsanity.com/paper/2503.05613/) (2025)
- [Understanding Reasoning in Thinking Language Models via Steering Vectors](https://mlsanity.com/paper/2506.18167/) (2025)
- [Steering Large Language Model Activations in Sparse Spaces](https://mlsanity.com/paper/2503.00177/) (2025)
- [Interpretable Steering of Large Language Models with Feature Guided Activation Additions](https://mlsanity.com/paper/2501.09929/) (2025)
- [SAIF: A Sparse Autoencoder Framework for Interpreting and Steering Instruction Following of Language Models](https://mlsanity.com/paper/2502.11356/) (2025)
- [SAE-SSV: Supervised Steering in Sparse Representation Spaces for Reliable Control of Language Models](https://mlsanity.com/paper/2505.16188/) (2025)
- [Understanding the Effect of Noise in LLM Training Data with Algorithmic Chains of Thought](https://mlsanity.com/paper/2402.04004/) (2024)
- [Differentially Private Steering for Large Language Model Alignment](https://mlsanity.com/paper/2501.18532/) (2025)
- [Uncovering Latent Chain of Thought Vectors in Language Models](https://mlsanity.com/paper/2409.14026/) (2024)
- [Improving LLM Reasoning through Interpretable Role-Playing Steering](https://mlsanity.com/paper/2506.07335/) (2025)
- [Denoising Concept Vectors with Sparse Autoencoders for Improved Language Model Steering](https://mlsanity.com/paper/2505.15038/) (2025)
- [Sparse Activation Editing for Reliable Instruction Following in Narratives](https://mlsanity.com/paper/2505.16505/) (2025)

## Built on by (most cited first, 10 of 10)

- [TACT: Mitigating Overthinking and Overacting in Coding Agents via Activation Steering](https://mlsanity.com/paper/2605.05980/) (2026)
- [Multilingual Steering by Design: Multilingual Sparse Autoencoders and Principled Layer Selection](https://mlsanity.com/paper/2605.23036/) (2026)
- [ORBIT: Training-Free Multi-Attribute Behavioral Steering via Orthogonal Subspace Rotation](https://mlsanity.com/paper/2606.22357/) (2026)
- [Graph-Regularized Sparse Autoencoders for LLM Safety Steering](https://mlsanity.com/paper/2512.06655/) (2025)
- [Mechanistic Indicators of Steering Effectiveness in Large Language Models](https://mlsanity.com/paper/2602.01716/) (2026)
- [Steered Generation via Gradient-Based Optimization on Sparse Query Features](https://mlsanity.com/paper/2605.23040/) (2026)
- [SAE-StatSteer: Statistical Consensus Feature Selection for Optimization-Free Activation Steering of Large Language Models](https://mlsanity.com/paper/2607.19364/) (2026)
- [Do Unified Multimodal Models Think in One Space? A Lens Through Cross-Branch Steering](https://mlsanity.com/paper/2607.26411/) (2026)
- [Key Path Identification for Resolving Knowledge Conflicts via SAE-based Steering](https://mlsanity.com/paper/2609.08173/) (2026)
- [Extracting Persona Subspaces Through Iterative Nullspace Projection For Modulation](https://mlsanity.com/paper/2610.04676/) (2026)

## Abstract (arXiv)

Steering has emerged as a promising approach in controlling large language models (LLMs) without modifying model parameters. However, most existing steering methods rely on large-scale datasets to learn clear behavioral information, which limits their applicability in many real-world scenarios. The steering vectors extracted from small dataset often contain task-irrelevant noising features, which degrades their effectiveness. To refine the steering vectors learned from limited data, we introduce Refinement of Steering Vector via Sparse Autoencoder (SAE-RSV) that leverages SAEs to semantically denoise and augment the steering vectors. In our framework, we first remove task-irrelevant features according to their semantics provided by SAEs, and then enrich task-relevant features missing from the small dataset through their semantic similarity to the identified relevant features. Extensive experiments demonstrate that the proposed SAE-RSV substantially outperforms all the baseline methods including supervised fine-tuning. Our findings show that effective steering vector can be constructed from limited training data by refining the original steering vector through SAEs.