mlsanityData to 14 Oct 2026

Enhancing LLM Steering through Sparse Autoencoder-Based Vector Refinement

arXiv 2509.23799 ↗First posted 28 Sep 2025cs.LG cs.AIcited by 10

On the map: Mechanistic interpretability → SAE features and their applications

Abstract (arXiv)

Steering has emerged as a promising approach in controlling large language models (LLMs) without modifying model parameters. However, most existing steering methods rely on large-scale datasets to learn clear behavioral information, which limits their applicability in many real-world scenarios. The steering vectors extracted from small dataset often contain task-irrelevant noising features, which degrades their effectiveness. To refine the steering vectors learned from limited data, we introduce Refinement of Steering Vector via Sparse Autoencoder (SAE-RSV) that leverages SAEs to semantically denoise and augment the steering vectors. In our framework, we first remove task-irrelevant features according to their semantics provided by SAEs, and then enrich task-relevant features missing from the small dataset through their semantic similarity to the identified relevant features. Extensive experiments demonstrate that the proposed SAE-RSV substantially outperforms all the baseline methods including supervised fine-tuning. Our findings show that effective steering vector can be constructed from limited training data by refining the original steering vector through SAEs.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in SAE features and their applications, one of 13 directions in Mechanistic interpretability. Builds on 27 of its 34 references that are in the map. First posted on arXiv 28 Sep 2025.

Builds on

27 of 34 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025SAEs Are Good for Steering -- If You Select the Right Features26 May
  2. 2025SAE-SSV: Supervised Steering in Sparse Representation Spaces for Reliable Control of Language Models22 May
  3. 2025Denoising Concept Vectors with Sparse Autoencoders for Improved Language Model Steering21 May
  4. 2025A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Models7 Mar
  5. 2025SAIF: A Sparse Autoencoder Framework for Interpreting and Steering Instruction Following of Language Models17 Feb
  6. 2025AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders28 Jan
  7. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Models15 Sep

Other references · 20

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  5. 2016Understanding intermediate layers using linear classifier probescited by 1,353
  6. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  7. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
  8. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  9. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
  10. 2025Open Problems in Mechanistic Interpretabilitycited by 179
  11. 2024Analyzing the Generalization and Reliability of Steering Vectorscited by 143
  12. 2024Do I Know This Entity? Knowledge Awareness and Hallucinations in Language Modelscited by 115
8 more
  1. 2025Understanding Reasoning in Thinking Language Models via Steering Vectorscited by 72
  2. 2025Steering Large Language Model Activations in Sparse Spacescited by 58
  3. 2025Interpretable Steering of Large Language Models with Feature Guided Activation Additionscited by 35
  4. 2024Understanding the Effect of Noise in LLM Training Data with Algorithmic Chains of Thoughtcited by 19
  5. 2025Differentially Private Steering for Large Language Model Alignmentcited by 15
  6. 2024Uncovering Latent Chain of Thought Vectors in Language Modelscited by 14
  7. 2025Improving LLM Reasoning through Interpretable Role-Playing Steeringcited by 12
  8. 2025Sparse Activation Editing for Reliable Instruction Following in Narrativescited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 9.

10 papers.

  1. 2026TACT: Mitigating Overthinking and Overacting in Coding Agents via Activation Steeringcited by 6
  2. 2026Multilingual Steering by Design: Multilingual Sparse Autoencoders and Principled Layer Selectioncited by 4
  3. 2026ORBIT: Training-Free Multi-Attribute Behavioral Steering via Orthogonal Subspace Rotationcited by 4
  4. 2025Graph-Regularized Sparse Autoencoders for LLM Safety Steeringcited by 1
  5. 2026Mechanistic Indicators of Steering Effectiveness in Large Language Modelscited by 0
  6. 2026Steered Generation via Gradient-Based Optimization on Sparse Query Featurescited by 0
  7. 2026SAE-StatSteer: Statistical Consensus Feature Selection for Optimization-Free Activation Steering of Large Language Modelscited by 0
  8. 2026Do Unified Multimodal Models Think in One Space? A Lens Through Cross-Branch Steeringcited by 0
  9. 2026Key Path Identification for Resolving Knowledge Conflicts via SAE-based Steeringcited by 0
  10. 2026Extracting Persona Subspaces Through Iterative Nullspace Projection For Modulationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Steering LLMs? Actually, Sparse Autoencoders can outperform simple baselines
  2. 2024Can sparse autoencoders be used to decompose and interpret steering vectors?
  3. 2025CorrSteer: Generation-Time LLM Steering via Correlated Sparse Autoencoder Features
  4. 2024Improving Steering Vectors by Targeting Sparse Autoencoder Features
  5. 2026Enhancing SAE-based Steering via Neighbor Integrated Feature Selection
  6. 2026Towards Steering without Sacrifice: Principled Training of Steering Vectors for Prompt-only Interventions
  7. 2026Disentangling Steering Vectors
  8. 2026Pre-Intervention Prediction of Sparse Autoencoder Steering Side Effects