# HyperSteer: Activation Steering at Scale with Hypernetworks

arXiv: https://arxiv.org/abs/2506.03292 · HTML: https://mlsanity.com/paper/2506.03292/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Placed in Activation steering methods, one of 13 directions in Mechanistic interpretability.
- Builds on 14 of its 22 references that are in the map.
- First posted on arXiv 3 Jun 2025.

## Placed in

- Mechanistic interpretability / [Activation steering methods](https://mlsanity.com/mechinterp/activation-steering-methods/)

## Builds on (most cited first, 14 of 14)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets](https://mlsanity.com/paper/2310.06824/) (2023)
- [Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models](https://mlsanity.com/paper/2403.19647/) (2024)
- [Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2](https://mlsanity.com/paper/2408.05147/) (2024)
- [AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders](https://mlsanity.com/paper/2501.17148/) (2025)
- [Natural Language Descriptions of Deep Visual Features](https://mlsanity.com/paper/2201.11114/) (2022)
- [Under the Hood: Using Diagnostic Classifiers to Investigate and Improve how Language Models Track Agreement Information](https://mlsanity.com/paper/1808.08079/) (2018)
- [pyvene: A Library for Understanding and Improving PyTorch Models via Interventions](https://mlsanity.com/paper/2403.07809/) (2024)
- [HyperTuning: Toward Adapting Large Language Models without Back-propagation](https://mlsanity.com/paper/2211.12485/) (2022)
- [HyperDAS: Towards Automating Mechanistic Interpretability with Hypernetworks](https://mlsanity.com/paper/2503.10894/) (2025)

## Built on by (most cited first, 23 of 23)

- [AlphaSteer: Learning Refusal Steering with Principled Null-Space Constraint](https://mlsanity.com/paper/2506.07022/) (2025)
- [Observing and Controlling Features in Vision-Language-Action Models](https://mlsanity.com/paper/2603.05487/) (2026)
- [COLD-Steer: Steering Large Language Models via In-Context One-step Learning Dynamics](https://mlsanity.com/paper/2603.06495/) (2026)
- [What Drives Representation Steering? A Mechanistic Case Study on Steering Refusal](https://mlsanity.com/paper/2604.08524/) (2026)
- [Steering Safely or Off a Cliff? Rethinking Specificity and Robustness in Inference-Time Interventions](https://mlsanity.com/paper/2602.06256/) (2026)
- [Beyond Steering Vector: Flow-based Activation Steering for Inference-Time Intervention](https://mlsanity.com/paper/2605.05892/) (2026)
- [Steer Like the LLM: Activation Steering that Mimics Prompting](https://mlsanity.com/paper/2605.03907/) (2026)
- [Evaluating Sparse Autoencoders for Monosemantic Representation](https://mlsanity.com/paper/2508.15094/) (2025)
- [How Controllable Are Large Language Models? A Unified Evaluation across Behavioral Granularities](https://mlsanity.com/paper/2603.02578/) (2026)
- [YaPO: Learnable Sparse Activation Steering Vectors for Domain Adaptation](https://mlsanity.com/paper/2601.08441/) (2026)
- [Towards Steering without Sacrifice: Principled Training of Steering Vectors for Prompt-only Interventions](https://mlsanity.com/paper/2605.05983/) (2026)
- [When Is Rank-1 Steering Cheap? Geometry, Granularity, and Budgeted Search](https://mlsanity.com/paper/2605.16362/) (2026)
- [Palette: A Modular, Controllable, and Efficient Framework for On-demand Authorized Safety Alignment Relaxation in LLMs](https://mlsanity.com/paper/2605.24154/) (2026)
- [Program-as-Weights: A Programming Paradigm for Fuzzy Functions](https://mlsanity.com/paper/2607.02512/) (2026)
- [Resist and Update: Counterfactual Report Coordinates for Incentive-Compatible LLMs](https://mlsanity.com/paper/2607.12985/) (2026)
- [Does Steering Break Your Model? A Multi-Dimensional Evaluation Suite for LLM Steering Methods](https://mlsanity.com/paper/2610.07722/) (2026)
- [AntiPaSTO: Self-Supervised Honesty Steering via Anti-Parallel Representations](https://mlsanity.com/paper/2601.07473/) (2026)
- [BarrierSteer: LLM Safety via Learning Barrier Steering](https://mlsanity.com/paper/2602.20102/) (2026)
- [HyperTransport: Amortized Conditioning of T2I Generative Models](https://mlsanity.com/paper/2605.08254/) (2026)
- [Activation Steering for Synthetic Data Generation: The Role of Diversity in Downstream Safety Detection](https://mlsanity.com/paper/2605.28664/) (2026)
- [Replay-Gated Neural Execution: Decoupling Persistent Behavioral Specifications from Neural Realizations in Frozen Language Models](https://mlsanity.com/paper/2609.22243/) (2026)
- [MetaSteer: Context-Conditioned, nonlinear Steering via Attention-Projection Adaptation](https://mlsanity.com/paper/2609.38718/) (2026)
- [Hesitation Has a Geometry: Entropy-Trained Hyperbolic Probes for Sparse Activation Steering](https://mlsanity.com/paper/2610.02391/) (2026)

## Abstract (arXiv)

Steering language models (LMs) by modifying internal activations is a popular approach for controlling text generation. Unsupervised dictionary learning methods, e.g., sparse autoencoders, can be scaled to produce many steering vectors, but lack guarantees on the individual efficacy of each vector and control over the coverage of relevant steering tasks. In contrast, supervised methods for constructing steering vectors are targeted and effective, but require more data collection and training for each additional steering vector produced. In this work, we introduce HyperSteer, a family of hypernetwork-based architectures which are trained end-to-end to generate steering vectors conditioned on the natural language steering prompts and the internals of the steered LM. In our evaluations, we show that scaling HyperSteer with thousands of steering prompts exceeds the performance of state-of-the-art activation steering methods, even on steering prompts never seen during training. Moreover, HyperSteer performs on par with steering-via-prompting.