mlsanityData to 14 Oct 2026

HyperSteer: Activation Steering at Scale with Hypernetworks

arXiv 2506.03292 ↗First posted 3 Jun 2025cs.CL cs.AI cs.LGcited by 23

On the map: Mechanistic interpretability → Activation steering methods

Abstract (arXiv)

Steering language models (LMs) by modifying internal activations is a popular approach for controlling text generation. Unsupervised dictionary learning methods, e.g., sparse autoencoders, can be scaled to produce many steering vectors, but lack guarantees on the individual efficacy of each vector and control over the coverage of relevant steering tasks. In contrast, supervised methods for constructing steering vectors are targeted and effective, but require more data collection and training for each additional steering vector produced. In this work, we introduce HyperSteer, a family of hypernetwork-based architectures which are trained end-to-end to generate steering vectors conditioned on the natural language steering prompts and the internals of the steered LM. In our evaluations, we show that scaling HyperSteer with thousands of steering prompts exceeds the performance of state-of-the-art activation steering methods, even on steering prompts never seen during training. Moreover, HyperSteer performs on par with steering-via-prompting.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Placed in Activation steering methods, one of 13 directions in Mechanistic interpretability. Builds on 14 of its 22 references that are in the map. First posted on arXiv 3 Jun 2025.

Builds on

14 of 22 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Steering Llama 2 via Contrastive Activation Addition9 Dec

Other references · 13

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  3. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  4. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  5. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 712
  6. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 435
  7. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 394
  8. 2025AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoderscited by 218
  9. 2022Natural Language Descriptions of Deep Visual Featurescited by 137
  10. 2018Under the Hood: Using Diagnostic Classifiers to Investigate and Improve how Language Models Track Agreement Informationcited by 97
  11. 2024pyvene: A Library for Understanding and Improving PyTorch Models via Interventionscited by 54
  12. 2022HyperTuning: Toward Adapting Large Language Models without Back-propagationcited by 46
1 more
  1. 2025HyperDAS: Towards Automating Mechanistic Interpretability with Hypernetworkscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 21.

23 papers.

Sort:

By area:

  1. 2025AlphaSteer: Learning Refusal Steering with Principled Null-Space Constraintcited by 49
  2. 2026Observing and Controlling Features in Vision-Language-Action Modelscited by 16
  3. 2026COLD-Steer: Steering Large Language Models via In-Context One-step Learning Dynamicscited by 8
  4. 2026What Drives Representation Steering? A Mechanistic Case Study on Steering Refusalcited by 7
  5. 2026Steering Safely or Off a Cliff? Rethinking Specificity and Robustness in Inference-Time Interventionscited by 6
  6. 2026Beyond Steering Vector: Flow-based Activation Steering for Inference-Time Interventioncited by 6
  7. 2026Steer Like the LLM: Activation Steering that Mimics Promptingcited by 5
  8. 2025Evaluating Sparse Autoencoders for Monosemantic Representationcited by 4
  9. 2026How Controllable Are Large Language Models? A Unified Evaluation across Behavioral Granularitiescited by 3
  10. 2026YaPO: Learnable Sparse Activation Steering Vectors for Domain Adaptationcited by 2
  11. 2026Towards Steering without Sacrifice: Principled Training of Steering Vectors for Prompt-only Interventionscited by 1
  12. 2026When Is Rank-1 Steering Cheap? Geometry, Granularity, and Budgeted Searchcited by 1
11 more
  1. 2026Palette: A Modular, Controllable, and Efficient Framework for On-demand Authorized Safety Alignment Relaxation in LLMscited by 1
  2. 2026Program-as-Weights: A Programming Paradigm for Fuzzy Functionscited by 1
  3. 2026Resist and Update: Counterfactual Report Coordinates for Incentive-Compatible LLMscited by 1
  4. 2026Does Steering Break Your Model? A Multi-Dimensional Evaluation Suite for LLM Steering Methodscited by 1
  5. 2026AntiPaSTO: Self-Supervised Honesty Steering via Anti-Parallel Representationscited by 0
  6. 2026BarrierSteer: LLM Safety via Learning Barrier Steeringcited by 0
  7. 2026HyperTransport: Amortized Conditioning of T2I Generative Modelscited by 0
  8. 2026Activation Steering for Synthetic Data Generation: The Role of Diversity in Downstream Safety Detectioncited by 0
  9. 2026Replay-Gated Neural Execution: Decoupling Persistent Behavioral Specifications from Neural Realizations in Frozen Language Modelscited by 0
  10. 2026MetaSteer: Context-Conditioned, nonlinear Steering via Attention-Projection Adaptationcited by 0
  11. 2026Hesitation Has a Geometry: Entropy-Trained Hyperbolic Probes for Sparse Activation Steeringcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Steering When Necessary: Flexible Steering Large Language Models with Backtracking
  2. 2026Disentangling Steering Vectors
  3. 2025CorrSteer: Generation-Time LLM Steering via Correlated Sparse Autoencoder Features
  4. 2025Interpretable Steering of Large Language Models with Feature Guided Activation Additions
  5. 2025Painless Activation Steering: An Automated, Lightweight Approach for Post-Training Large Language Models
  6. 2026Contextual Linear Activation Steering of Language Models
  7. 2024Activation Scaling for Steering and Interpreting Language Models
  8. 2023Improving Activation Steering in Language Models with Mean-Centring