mlsanityData to 14 Oct 2026

SAE-SSV: Supervised Steering in Sparse Representation Spaces for Reliable Control of Language Models

arXiv 2505.16188 ↗First posted 22 May 2025cs.CLcited by 29

On the map: Mechanistic interpretability → SAE features and their applications

Abstract (arXiv)

Large language models (LLMs) have demonstrated impressive capabilities in natural language understanding and generation, but controlling their behavior reliably remains challenging, especially in open-ended generation settings. This paper introduces a novel supervised steering approach that operates in sparse, interpretable representation spaces. We employ sparse autoencoders (SAEs) to obtain sparse latent representations that aim to disentangle semantic attributes from model activations. Then we train linear classifiers to identify a small subspace of task-relevant dimensions in latent representations. Finally, we learn supervised steering vectors constrained to this subspace, optimized to align with target behaviors. Experiments across sentiment, truthfulness, and political polarity steering tasks with multiple LLMs demonstrate that our supervised steering vectors achieve higher success rates with minimal degradation in generation quality compared to existing methods. Further analysis reveals that a notably small subspace is sufficient for effective steering, enabling more targeted and interpretable interventions. Our implementation is publicly available at https://github.com/Ineedanamehere/SAE-SSV.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Placed in SAE features and their applications, one of 13 directions in Mechanistic interpretability. Builds on 33 of its 41 references that are in the map. First posted on arXiv 22 May 2025.

Builds on

33 of 41 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Are Sparse Autoencoders Useful? A Case Study in Sparse Probing23 Feb
  2. 2025Interpreting and Steering LLMs with Mutual Information-based Explanations on Sparse Autoencoders21 Feb
  3. 2025Sparse Autoencoder Features for Classifications and Transferability17 Feb
  4. 2025SAIF: A Sparse Autoencoder Framework for Interpreting and Steering Instruction Following of Language Models17 Feb
  5. 2025AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders28 Jan
  6. 2024Improving Steering Vectors by Targeting Sparse Autoencoder Features4 Nov
  7. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Models15 Sep

Other references · 26

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  3. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,319
  4. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  5. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
  6. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  7. 2023Steering Language Models With Activation Engineeringcited by 922
  8. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
  9. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 799
  10. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 712
  11. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 609
  12. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 394
14 more
  1. 2024Do I Know This Entity? Knowledge Awareness and Hallucinations in Language Modelscited by 115
  2. 2024Adaptive Activation Steering: A Tuning-Free LLM Truthfulness Improvement Method for Diverse Hallucinations Categoriescited by 77
  3. 2023Word Embeddings Are Steers for Language Modelscited by 70
  4. 2023Improving Activation Steering in Language Models with Mean-Centringcited by 68
  5. 2024Steering Away from Harm: An Adaptive Approach to Defending Vision Language Model Against Jailbreakscited by 48
  6. 2025Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understandingcited by 47
  7. 2025A Unified Understanding and Evaluation of Steering Methodscited by 46
  8. 2025FairSteer: Inference Time Debiasing for LLMs with Dynamic Activation Steeringcited by 40
  9. 2024On the Relationship between Truth and Political Bias in Language Modelscited by 31
  10. 2024Controlling Large Language Model Agents with Entropic Activation Steeringcited by 29
  11. 2024Beyond Single Concept Vector: Modeling Concept Subspace in LLMs with Gaussian Distributioncited by 27
  12. 2023Efficient fair PCA for fair representation learningcited by 23
  13. 2024Text Generation: A Systematic Literature Review of Tasks, Evaluation, and Challengescited by 18
  14. 2024Evaluating the Smooth Control of Attribute Intensity in Text Generation with LLMscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 23.

29 papers.

Sort:

By area:

  1. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 27
  2. 2025Enhancing LLM Steering through Sparse Autoencoder-Based Vector Refinementcited by 10
  3. 2025SAGE: An Agentic Explainer Framework for Interpreting SAE Features in Language Modelscited by 10
  4. 2026All Circuits Lead to Rome: Rethinking Functional Anisotropy in Circuit and Sheaf Discovery for LLMscited by 9
  5. 2025Evaluating Sparse Autoencoders for Monosemantic Representationcited by 4
  6. 2026Farther the Shift, Sparser the Representation: Analyzing OOD Mechanisms in LLMscited by 4
  7. 2025Quantifying Conversation Drift in MCP via Latent Polytopecited by 2
  8. 2025SDA: Steering-Driven Distribution Alignment for Open LLMs without Fine-Tuningcited by 2
  9. 2026YaPO: Learnable Sparse Activation Steering Vectors for Domain Adaptationcited by 2
  10. 2026Exploring Diverse Generation Paths via Inference-time Stiefel Activation Steeringcited by 2
  11. 2026Why Are Some Emotions Harder for LLMs? Uncovering the Causal Mechanisms of Emotion Inference via Sparse Autoencoderscited by 1
  12. 2026Where Concept Erasure Should Occur: Concept-Layer Alignment in Text-to-Video Diffusion Modelscited by 1
17 more
  1. 2026Position: AI Safety Requires Effective Controllabilitycited by 1
  2. 2026SAEExplainer: Interpreting SAE Features with Activation-Guided Preference Optimizationcited by 1
  3. 2026CircuitSteer: Geometrically Aligned Multi-Layer Steering via Sparse Autoencoder Circuitscited by 1
  4. 2026REINS: Refusal-Enhanced Inhibitory Steering with Sparse Autoencoder Featurescited by 1
  5. 2026Enhancing SAE-based Steering via Neighbor Integrated Feature Selectioncited by 1
  6. 2025Bridging Compositional and Distributional Semantics: A Survey on Latent Semantic Geometry via AutoEncodercited by 0
  7. 2026Facet-Level Persona Control by Trait-Activated Routing with Contrastive SAE for Role-Playing LLMscited by 0
  8. 2026Steered Generation via Gradient-Based Optimization on Sparse Query Featurescited by 0
  9. 2026Persistent Sparse Autoencoders: Learning Feature-Specific Timescales in Language Model Representationscited by 0
  10. 2026SAE-StatSteer: Statistical Consensus Feature Selection for Optimization-Free Activation Steering of Large Language Modelscited by 0
  11. 2026Measuring Semantic Abstractness of SAE Features via Nonlocalitycited by 0
  12. 2026EXPOSE: Explainable and Domain-Robust Embeddings from Pathology Vision Foundation Models using Sparse Autoencoderscited by 0
  13. 2026IDEEA: training-free Input-Dependent stEEring via Activation cluster matchingcited by 0
  14. 2026When Steering Fails in Latent Reasoning: A Latent-to-Language Transition Gapcited by 0
  15. 2026STR: Supervised Transcoder Replacement for Reducing Steering Side Effectscited by 0
  16. 2026MetaSteer: Context-Conditioned, nonlinear Steering via Attention-Projection Adaptationcited by 0
  17. 2026SelectTTRL: Representation-Guided Data Selection for Efficient Test-Time Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Steering LLMs? Actually, Sparse Autoencoders can outperform simple baselines
  2. 2025Steering Large Language Model Activations in Sparse Spaces
  3. 2025A Comparative Analysis of Sparse Autoencoder and Activation Difference in Language Model Steering
  4. 2024Can sparse autoencoders be used to decompose and interpret steering vectors?
  5. 2025Interpretable LLM Guardrails via Sparse Representation Steering
  6. 2026Multilingual Steering by Design: Multilingual Sparse Autoencoders and Principled Layer Selection
  7. 2022Extracting Latent Steering Vectors from Pretrained Language Models
  8. 2025Denoising Concept Vectors with Sparse Autoencoders for Improved Language Model Steering