mlsanityData to 14 Oct 2026

Denoising Concept Vectors with Sparse Autoencoders for Improved Language Model Steering

arXiv 2505.15038 ↗First posted 21 May 2025cs.CL cs.AIcited by 9

On the map: Mechanistic interpretability → SAE features and their applications

Abstract (arXiv)

Linear concept vectors effectively steer LLMs, but existing methods suffer from noisy features in diverse datasets that undermine steering robustness. We propose Sparse Autoencoder-Denoised Concept Vectors (SDCV), which selectively keep the most discriminative SAE latents while reconstructing hidden representations. Our key insight is that concept-relevant signals can be explicitly separated from dataset noise by scaling up activations of top-k latents that best differentiate positive and negative samples. Applied to linear probing and difference-in-mean, SDCV consistently improves steering success rates by 4-16% across six challenging concepts, while maintaining topic relevance.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in SAE features and their applications, one of 13 directions in Mechanistic interpretability. Builds on 15 of its 23 references that are in the map. First posted on arXiv 21 May 2025.

Builds on

15 of 23 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025SAIF: A Sparse Autoencoder Framework for Interpreting and Steering Instruction Following of Language Models17 Feb
  2. 2025AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders28 Jan
  3. 2024A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders22 Sep

Other references · 12

  1. 2017Attention Is All You Needcited by 60,361
  2. 2016Understanding intermediate layers using linear classifier probescited by 1,353
  3. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,319
  4. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  5. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
  6. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 712
  7. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 394
  8. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 245
  9. 2024Improving Instruction-Following in Language Models through Activation Steeringcited by 160
  10. 2023Improving Activation Steering in Language Models with Mean-Centringcited by 68
  11. 2024Beyond Single Concept Vector: Modeling Concept Subspace in LLMs with Gaussian Distributioncited by 27
  12. 2024Uncovering Latent Chain of Thought Vectors in Language Modelscited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 7.

9 papers.

  1. 2025Enhancing LLM Steering through Sparse Autoencoder-Based Vector Refinementcited by 10
  2. 2026From Weights to Activations: Is Steering the Next Frontier of Adaptation?cited by 8
  3. 2026The Cylindrical Representation Hypothesis for Language Model Steeringcited by 5
  4. 2026Multilingual Steering by Design: Multilingual Sparse Autoencoders and Principled Layer Selectioncited by 4
  5. 2025Graph-Regularized Sparse Autoencoders for LLM Safety Steeringcited by 1
  6. 2026Facet-Level Persona Control by Trait-Activated Routing with Contrastive SAE for Role-Playing LLMscited by 0
  7. 2026α-TCAV: A Unified Framework for Testing with Concept Activation Vectorscited by 0
  8. 2026Steered Generation via Gradient-Based Optimization on Sparse Query Featurescited by 0
  9. 2026SAEVerbalizer: Generating Explanations for Sparse Autoencoder Features via Representation Verbalizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Steering LLMs? Actually, Sparse Autoencoders can outperform simple baselines
  2. 2025SAE-SSV: Supervised Steering in Sparse Representation Spaces for Reliable Control of Language Models
  3. 2025Calibrating Lightweight Sparse Autoencoder Feature Steering
  4. 2024Can sparse autoencoders be used to decompose and interpret steering vectors?
  5. 2025Interpretable and Steerable Concept Bottleneck Sparse Autoencoders
  6. 2024Scaling and evaluating sparse autoencoders
  7. 2025CorrSteer: Generation-Time LLM Steering via Correlated Sparse Autoencoder Features
  8. 2025Are Sparse Autoencoders Useful? A Case Study in Sparse Probing