When Is Rank-1 Steering Cheap? Geometry, Granularity, and Budgeted Search
Not on the map. Why?
Abstract (arXiv)
Activation steering offers a lightweight way to control LLMs without retraining, but its effectiveness varies sharply across concepts. Prior work often reads this variability as evidence that many concepts are not captured by a single steering direction. We argue instead that much of it reflects search difficulty: a useful rank-1 intervention often exists, but finding it can be expensive. We formalize rank-1 steering as a budget-constrained optimization over intervention layer and coefficient. Across concepts and model families, prompt-boundary directional alignment predicts where effective interventions occur, enabling geometry-guided search that reaches high utility with substantially fewer evaluations, reducing the trials needed to recover 95% of best-found utility by 39.8% on average across three model families. To explain why some concepts remain expensive even under better search, we introduce concept granularity, a measure of directional heterogeneity across contrastive contexts. Granularity distinguishes concepts whose difference vectors share a stable global direction from those where prompts agree locally within each input but the utility-maximizing direction rotates systematically across inputs. Higher granularity is associated with slower convergence and lower best-found performance (Pearson r=0.44 with trials-to-95%, r=-0.46 with best-found utility, both p<0.001). We present GRACE, a Granularity- and Representation-Aware Concept Engineering framework that uses activation geometry to diagnose the dominant source of steering difficulty, select the appropriate remedy, and allocate optimization effort efficiently. Our results shift the frame from "when does rank-1 fail?" to "when is rank-1 cheap and stable?", turning activation geometry from a descriptive tool into an actionable prior for LLM control.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 17 of its 21 references that are in the map. First posted on arXiv 9 May 2026.
Builds on
17 of 21 references.
References · 17
- 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
- 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,009
- 2023Steering Language Models With Activation Engineeringcited by 824
- 2025Persona Vectors: Monitoring and Controlling Character Traits in Language Modelscited by 303
- 2025AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoderscited by 199
- 2024Programming Refusal with Conditional Activation Steeringcited by 191
- 2024Improving Instruction-Following in Language Models through Activation Steeringcited by 149
- 2024Personalized Steering of Large Language Models: Versatile Steering Vectors Through Bi-directional Preference Optimizationcited by 112
- 2024Steering Language Model Refusal with Sparse Autoencoderscited by 89
- 2025A Unified Understanding and Evaluation of Steering Methodscited by 40
- 2025Understanding (Un)Reliability of Steering Vectors in Language Modelscited by 39
- 2025HyperSteer: Activation Steering at Scale with Hypernetworkscited by 20
5 more
- 2025Improved Representation Steering for Language Modelscited by 17
- 2025Steering LLM Reasoning Through Bias-Only Adaptationcited by 16
- 2026Steering Vector Fields for Context-Aware Inference-Time Control in Large Language Modelscited by 10
- 2025What Can We Actually Steer? A Multi-Behavior Study of Activation Controlcited by 6
- 2026Persona Vectors in Games: Measuring and Steering Strategies via Activation Vectorscited by 6
Built on by
Citing papers per year: 2026: 1.
1 paper.