mlsanityData to 14 Oct 2026

CorrSteer: Generation-Time LLM Steering via Correlated Sparse Autoencoder Features

arXiv 2508.12535 ↗First posted 18 Aug 2025cs.CL cs.AI cs.LGcited by 12

On the map: Mechanistic interpretability → SAE features and their applications

Abstract (arXiv)

Sparse Autoencoders (SAEs) can extract interpretable features from large language models (LLMs) without supervision. However, their effectiveness in downstream steering tasks is limited by the requirement for contrastive datasets or large activation storage. To address these limitations, we propose CorrSteer, which selects features by correlating sample correctness with SAE activations from generated tokens at inference time. This approach uses only inference-time activations to extract more relevant features, thereby reducing spurious correlations. It also obtains steering coefficients from average activations, automating the entire pipeline. Our method shows improved task performance on QA, bias mitigation, jailbreaking prevention, and reasoning benchmarks on Gemma-2 2B and LLaMA-3.1 8B, notably achieving a +3.3% improvement in MMLU performance with 4000 samples and a +27.2% improvement in HarmBench with only 108 samples. Selected features demonstrate semantically meaningful patterns aligned with each task's requirements, revealing the underlying capabilities that drive performance. Our work establishes correlation-based selection as an effective and scalable approach for automated SAE steering across language model applications.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Placed in SAE features and their applications, one of 13 directions in Mechanistic interpretability. Builds on 36 of its 52 references that are in the map. First posted on arXiv 18 Aug 2025.

Builds on

36 of 52 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025Model Unlearning via Sparse Autoencoder Subspace Guided Projections30 May
  2. 2025SAEs Are Good for Steering -- If You Select the Right Features26 May
  3. 2025SAEs Can Improve Unlearning: Dynamic Sparse Autoencoder Guardrails for Precision Unlearning in LLMs11 Apr
  4. 2024Improving Steering Vectors by Targeting Sparse Autoencoder Features4 Nov
  5. 2024Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoders27 Oct
  6. 2024Steering Knowledge Selection Behaviours in LLMs via SAE-Based Representation Engineering21 Oct
  7. 2024Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders19 Jul
  8. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Models15 Sep

Other references · 28

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  6. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  7. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  8. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
  9. 2022Toy Models of Superpositioncited by 971
  10. 2023Steering Language Models With Activation Engineeringcited by 922
  11. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 724
  12. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 716
16 more
  1. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 712
  2. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 701
  3. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 505
  4. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 435
  5. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 394
  6. 2024Measuring short-form factuality in large language modelscited by 373
  7. 2022Extracting Latent Steering Vectors from Pretrained Language Modelscited by 225
  8. 2024Analyzing the Generalization and Reliability of Steering Vectorscited by 143
  9. 2024Steering Language Model Refusal with Sparse Autoencoderscited by 95
  10. 2025SAeUron: Interpretable Concept Unlearning in Diffusion Models with Sparse Autoencoderscited by 80
  11. 2015Sparse Overcomplete Word Vector Representationscited by 44
  12. 2025Interpretable Steering of Large Language Models with Feature Guided Activation Additionscited by 35
  13. 2024PaCE: Parsimonious Concept Engineering for Large Language Modelscited by 19
  14. 2024HEARTS: A Holistic Framework for Explainable, Sustainable and Robust Text Stereotype Detectioncited by 6
  15. 2025Circuit Stability Characterizes Language Model Generalizationcited by 6
  16. 2025Evaluating Neuron Explanations: A Unified Framework with Sanity Checkscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 12.

12 papers.

  1. 2026Steering Without Breaking: Mechanistically Informed Interventions for Discrete Diffusion Language Modelscited by 3
  2. 2026Correctness-Optimized Residual Activation Lens (CORAL): Transferrable and Calibration-Aware Inference-Time Steeringcited by 2
  3. 2026High-Dimensional Random Projection for Activation Steering in Language Modelscited by 1
  4. 2026REINS: Refusal-Enhanced Inhibitory Steering with Sparse Autoencoder Featurescited by 1
  5. 2026Enhancing SAE-based Steering via Neighbor Integrated Feature Selectioncited by 1
  6. 2026MoRFI: Monotonic Sparse Autoencoder Feature Identificationcited by 0
  7. 2026Steered Generation via Gradient-Based Optimization on Sparse Query Featurescited by 0
  8. 2026Toward Latent Language Model Skills Steering and Optimization: An Empirical Studycited by 0
  9. 2026Why Are LLM Backdoor Defenses Fragmented? A Feature-Level Explanation with Sparse Autoencoderscited by 0
  10. 2026SAEScientist-Bench: Can AI Agents Conduct Autonomous SAE Interpretability Research?cited by 0
  11. 2026MetaSteer: Context-Conditioned, nonlinear Steering via Attention-Projection Adaptationcited by 0
  12. 2026What Do Sparse Autoencoder Feature Discovery Scores Identify? From Activations to Interventionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Enhancing LLM Steering through Sparse Autoencoder-Based Vector Refinement
  2. 2026Steering LLMs? Actually, Sparse Autoencoders can outperform simple baselines
  3. 2026Multilingual Steering by Design: Multilingual Sparse Autoencoders and Principled Layer Selection
  4. 2025SAE-SSV: Supervised Steering in Sparse Representation Spaces for Reliable Control of Language Models
  5. 2026Strengthening Target-Language Features: SAE-Based Steering for Multilingual Inference
  6. 2025Interpreting and Steering LLMs with Mutual Information-based Explanations on Sparse Autoencoders
  7. 2025A Comparative Analysis of Sparse Autoencoder and Activation Difference in Language Model Steering
  8. 2026SAE-StatSteer: Statistical Consensus Feature Selection for Optimization-Free Activation Steering of Large Language Models