mlsanityData to 14 Oct 2026

High-Dimensional Random Projection for Activation Steering in Language Models

arXiv 2606.15092 ↗First posted 13 Jun 2026cs.LGcited by 1

On the map: Mechanistic interpretability → Activation steering methods

Abstract (arXiv)

Activation steering has emerged as a key methodology for controlling the behavior of large language models (LLMs). Existing difference-in-means based methods, however, are fundamentally limited: they capture only mean differences between class activations and fail to recover discriminative signals that naturally exist in the nonlinear feature subspace under the superposition hypothesis. Motivated by that, we propose High-Dimensional Random-projection for Activation Steering (HiDRA), a training-free approach that integrates seamlessly with existing activation steering methods. By performing activation addition in the projected high-dimensional space, HiDRA can provably capture a better discriminative structure beyond the reach of linear methods. Experiments across diverse LLM families and benchmarks demonstrate that HiDRA consistently outperforms baseline counterparts, achieving stronger behavioral control without significant computational overhead.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Activation steering methods, one of 13 directions in Mechanistic interpretability. Builds on 40 of its 50 references that are in the map. First posted on arXiv 13 Jun 2026.

Builds on

40 of 50 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Steering Large Language Model Activations in Sparse Spaces28 Feb
  2. 2023Steering Llama 2 via Contrastive Activation Addition9 Dec
  3. 2023Steering Language Models With Activation Engineering20 Aug

Other references · 37

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  6. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  7. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  8. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  9. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  10. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  11. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  12. 2022Red Teaming Language Models with Language Modelscited by 1,026
25 more
  1. 2022Toy Models of Superpositioncited by 971
  2. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
  3. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 716
  4. 2023A Comprehensive Overview of Large Language Modelscited by 648
  5. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 536
  6. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 451
  7. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 422
  8. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 331
  9. 2024tinyBenchmarks: evaluating LLMs with fewer examplescited by 310
  10. 2024Programming Refusal with Conditional Activation Steeringcited by 206
  11. 2026Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnetcited by 191
  12. 2016Linear Algebraic Structure of Word Senses, with Applications to Polysemycited by 165
  13. 2015On the Equivalence between Kernel Quadrature Rules and Random Feature Expansionscited by 138
  14. 2025Are Sparse Autoencoders Useful? A Case Study in Sparse Probingcited by 115
  15. 2024Steering Language Model Refusal with Sparse Autoencoderscited by 95
  16. 2025Understanding Reasoning in Thinking Language Models via Steering Vectorscited by 72
  17. 2024Controlling Language and Diffusion Models by Transporting Activationscited by 68
  18. 2025Angular Steering: Behavior Control via Rotation in Activation Spacecited by 53
  19. 2025The Hidden Dimensions of LLM Alignment: A Multi-Dimensional Analysis of Orthogonal Safety Directionscited by 41
  20. 2025SAIF: A Sparse Autoencoder Framework for Interpreting and Steering Instruction Following of Language Modelscited by 34
  21. 2025Taxonomy, Opportunities, and Challenges of Representation Engineering for Large Language Modelscited by 34
  22. 2025CorrSteer: Generation-Time LLM Steering via Correlated Sparse Autoencoder Featurescited by 12
  23. 2025Safety Subspaces are Not Linearly Distinct: A Fine-Tuning Case Studycited by 6
  24. 2026Stop Probing, Start Coding: Why Linear Probes and Sparse Autoencoders Fail at Compositional Generalisationcited by 6
  25. 2025A Comparative Analysis of Sparse Autoencoder and Activation Difference in Language Model Steeringcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Does Steering Break Your Model? A Multi-Dimensional Evaluation Suite for LLM Steering Methodscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Interpretable Steering of Large Language Models with Feature Guided Activation Additions
  2. 2026On the Non-Identifiability of Steering Vectors in Large Language Models
  3. 2026Beyond Linear Activation Steering: Invertible Latent Transformations for Controlling LLM Behavior
  4. 2026Disentangling Steering Vectors
  5. 2026Mechanistic Indicators of Steering Effectiveness in Large Language Models
  6. 2026COLD-Steer: Steering Large Language Models via In-Context One-step Learning Dynamics
  7. 2026Fine-Grained Activation Steering: Steering Less, Achieving More
  8. 2026Cross-Lingual Activation Steering for Multilingual Language Models