mlsanityData to 14 Oct 2026

Steering Large Language Model Activations in Sparse Spaces

arXiv 2503.00177 ↗First posted 28 Feb 2025cs.LG cs.AIcited by 58

On the map: Mechanistic interpretability → Activation steering methods

Abstract (arXiv)

A key challenge in AI alignment is guiding large language models (LLMs) to follow desired behaviors at test time. Activation steering, which modifies internal model activations during inference, offers a potential solution. However, prior work in dense activation spaces struggles with superposition, wherein multiple features become entangled, limiting interpretability and precise control. In contrast, sparse representations provide an untapped opportunity for more interpretable behavior modulation. In this work, we introduce sparse activation steering (SAS), a method that leverages sparse autoencoders (SAEs) to steer LLM behavior in sparse spaces. By isolating behavior-specific features through a contrastive prompt-pairing approach, we define a set of features that can selectively reinforce or suppress behaviors. Experiments on Gemma 2 LLMs show that SAS vectors enable nuanced behavioral modulation and finer-grained control. Furthermore, scaling SAEs improves monosemanticity of SAS vectors, suggesting more reliable and interpretable interventions.

Key facts. Cited by 58 papers in the map (data through 14 Oct 2026). Placed in Activation steering methods, one of 13 directions in Mechanistic interpretability. First posted on arXiv 28 Feb 2025.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 29; 2026: 29.

58 papers.

Sort:

By area:

  1. 2025Angular Steering: Behavior Control via Rotation in Activation Spacecited by 53
  2. 2025Navigate the Unknown: Enhancing LLM Reasoning with Intrinsic Motivation Guided Explorationcited by 34
  3. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 27
  4. 2025Beyond Prompt Engineering: Robust Behavior Control in LLMs via Steering Target Atomscited by 24
  5. 2025The Rogue Scalpel: Activation Steering Compromises LLM Safetycited by 21
  6. 2026Qwen-Scope: Turning Sparse Features into Development Tools for Large Language Modelscited by 21
  7. 2025Robust Hallucination Detection in LLMs via Adaptive Token Selectioncited by 19
  8. 2025Adaptive Multi-Subspace Representation Steering for Attribute Alignment in Large Language Modelscited by 16
  9. 2025Activation Steering with a Feedback Controllercited by 15
  10. 2025PIXEL: Adaptive Steering Via Position-wise Injection with eXact Estimated Levels under Subspace Calibrationcited by 11
  11. 2025Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safetycited by 10
  12. 2025Enhancing LLM Steering through Sparse Autoencoder-Based Vector Refinementcited by 10
38 more
  1. 2025Impatient Users Confuse AI Agents: High-fidelity Simulations of Human Traits for Testing Agentscited by 10
  2. 2025SAEdit: Token-level control for continuous image editing via Sparse AutoEncodercited by 9
  3. 2025STEER-BENCH: A Benchmark for Evaluating the Steerability of Large Language Modelscited by 8
  4. 2025Mechanistic Indicators of Understanding in Large Language Modelscited by 8
  5. 2026From Weights to Activations: Is Steering the Next Frontier of Adaptation?cited by 8
  6. 2025EasyEdit2: An Easy-to-use Steering Framework for Editing Large Language Modelscited by 7
  7. 2025Can LLMs Lie? Investigation beyond Hallucinationcited by 7
  8. 2025Resa: Transparent Reasoning Models via SAEscited by 6
  9. 2025AI shares emotion with humans across languages and culturescited by 6
  10. 2025Sparse Feature Coactivation Reveals Causal Semantic Modules in Large Language Modelscited by 6
  11. 2026RISER: Orchestrating Latent Reasoning Skills for Adaptive Activation Steeringcited by 6
  12. 2026Sparse Autoencoders are Capable LLM Jailbreak Mitigatorscited by 6
  13. 2025The Anatomy of Alignment: Decomposing Preference Optimization by Steering Sparse Featurescited by 5
  14. 2026Compositional Steering of Large Language Models with Steering Tokenscited by 5
  15. 2026Global Evolutionary Steering: Refining Activation Steering Control via Cross-Layer Consistencycited by 5
  16. 2026Wiring the 'Why': A Unified Taxonomy and Survey of Abductive Reasoning in LLMscited by 5
  17. 2026Perplexity Can Miss SAE Feature Damage Under Quantizationcited by 5
  18. 2025Evaluating Sparse Autoencoders for Monosemantic Representationcited by 4
  19. 2025AlignSAE: Concept-Aligned Sparse Autoencoderscited by 4
  20. 2026Multilingual Steering by Design: Multilingual Sparse Autoencoders and Principled Layer Selectioncited by 4
  21. 2025We Think, Therefore We Align LLMs to Helpful, Harmless and Honest Before They Go Wrongcited by 3
  22. 2025Toward Preference-aligned Large Language Models via Residual-based Model Steeringcited by 3
  23. 2025Task Vectors, Learned Not Extracted: Performance Gains and Mechanistic Insightcited by 3
  24. 2026How Controllable Are Large Language Models? A Unified Evaluation across Behavioral Granularitiescited by 3
  25. 2026Don't Lose Focus: Activation Steering via Key-Orthogonal Projectionscited by 3
  26. 2026Mechanism Design Is Not Enough: Prosocial Agents for Cooperative AIcited by 3
  27. 2025Beyond Interpretability: When, Why, and How Sparse Autoencoders Enable Label-Free Visual Steeringcited by 2
  28. 2025Quantifying Conversation Drift in MCP via Latent Polytopecited by 2
  29. 2025LatentGuard: Controllable Latent Steering for Robust Refusal of Attacks and Reliable Response Generationcited by 2
  30. 2025Painless Activation Steering: An Automated, Lightweight Approach for Post-Training Large Language Modelscited by 2
  31. 2026YaPO: Learnable Sparse Activation Steering Vectors for Domain Adaptationcited by 2
  32. 2026Tapered Language Modelscited by 2
  33. 2026From Refusal Tokens to Refusal Control: Discovering and Steering Category-Specific Refusal Directionscited by 1
  34. 2026DSPA: Dynamic SAE Steering for Data-Efficient Preference Alignmentcited by 1
  35. 2026SteeringDiffusion: A Bottlenecked Activation Control Interface for Diffusion Modelscited by 1
  36. 2026High-Dimensional Random Projection for Activation Steering in Language Modelscited by 1
  37. 2026Continuous Interpretive Steering for Scalar Diversitycited by 0
  38. 2026A Systematic Study of Training-Free Methods for Trustworthy Large Language Modelscited by 0

50 most cited of 58 All 58 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026A Coin Flip Per Token: Bernoulli Sparse Steering of Large Language Models
  2. 2025Interpretable Steering of Large Language Models with Feature Guided Activation Additions
  3. 2025SAE-SSV: Supervised Steering in Sparse Representation Spaces for Reliable Control of Language Models
  4. 2025A Comparative Analysis of Sparse Autoencoder and Activation Difference in Language Model Steering
  5. 2026CircuitSteer: Geometrically Aligned Multi-Layer Steering via Sparse Autoencoder Circuits
  6. 2026Beyond Linear Activation Steering: Invertible Latent Transformations for Controlling LLM Behavior
  7. 2025Interpretable LLM Guardrails via Sparse Representation Steering
  8. 2026SAE-StatSteer: Statistical Consensus Feature Selection for Optimization-Free Activation Steering of Large Language Models