# Semantics-Adaptive Activation Intervention for LLMs via Dynamic Steering Vectors

arXiv: https://arxiv.org/abs/2410.12299 · HTML: https://mlsanity.com/paper/2410.12299/

## Key facts

- Cited by 63 papers in the map (data through 14 Oct 2026).
- Placed in Activation steering methods, one of 13 directions in Mechanistic interpretability.
- Builds on 29 of its 39 references that are in the map.
- First posted on arXiv 16 Oct 2024.

## Placed in

- Mechanistic interpretability / [Activation steering methods](https://mlsanity.com/mechinterp/activation-steering-methods/)

## Builds on (most cited first, 29 of 29)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Representation Engineering: A Top-Down Approach to AI Transparency](https://mlsanity.com/paper/2310.01405/) (2023)
- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [Enhancing Chat Language Models by Scaling High-quality Instructional Conversations](https://mlsanity.com/paper/2305.14233/) (2023)
- [The Flan Collection: Designing Data and Methods for Effective Instruction Tuning](https://mlsanity.com/paper/2301.13688/) (2023)
- [The Linear Representation Hypothesis and the Geometry of Large Language Models](https://mlsanity.com/paper/2311.03658/) (2023)
- [The Falcon Series of Open Language Models](https://mlsanity.com/paper/2311.16867/) (2023)
- [ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection](https://mlsanity.com/paper/2203.09509/) (2022)
- [A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stories](https://mlsanity.com/paper/1604.01696/) (2016)
- [XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning](https://mlsanity.com/paper/2005.00333/) (2020)
- [In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steering](https://mlsanity.com/paper/2311.06668/) (2023)
- [Extracting Latent Steering Vectors from Pretrained Language Models](https://mlsanity.com/paper/2205.05124/) (2022)
- [How do Large Language Models Handle Multilingualism?](https://mlsanity.com/paper/2402.18815/) (2024)
- [Inspecting and Editing Knowledge Representations in Language Models](https://mlsanity.com/paper/2304.00740/) (2023)
- [Truth Forest: Toward Multi-Scale Truthfulness in Large Language Models through Intervention without Tuning](https://mlsanity.com/paper/2312.17484/) (2023)
- [Sharing Matters: Analysing Neurons Across Languages and Tasks in LLMs](https://mlsanity.com/paper/2406.09265/) (2024)
- [Retrieval-augmented Multilingual Knowledge Editing](https://mlsanity.com/paper/2312.13040/) (2023)
- [Interpreting and Exploiting Functional Specialization in Multi-Head Attention under Multi-task Learning](https://mlsanity.com/paper/2310.10318/) (2023)

## Built on by (most cited first, 30 of 63)

- [Angular Steering: Behavior Control via Rotation in Activation Space](https://mlsanity.com/paper/2510.26243/) (2025)
- [Representation Engineering for Large-Language Models: Survey and Research Challenges](https://mlsanity.com/paper/2502.17601/) (2025)
- [Spherical Steering: Geometry-Aware Activation Rotation for Language Models](https://mlsanity.com/paper/2602.08169/) (2026)
- [ODESteer: A Unified ODE-Based Steering Framework for LLM Alignment](https://mlsanity.com/paper/2602.17560/) (2026)
- [LLM-CAS: Dynamic Neuron Perturbation for Real-Time Hallucination Correction](https://mlsanity.com/paper/2512.18623/) (2025)
- [KV Cache Steering for Controlling Frozen LLMs](https://mlsanity.com/paper/2507.08799/) (2025)
- [Steering When Necessary: Flexible Steering Large Language Models with Backtracking](https://mlsanity.com/paper/2508.17621/) (2025)
- [Dynamic Multimodal Activation Steering for Hallucination Mitigation in Large Vision-Language Models](https://mlsanity.com/paper/2602.21704/) (2026)
- [Latent-space adversarial training with post-aware calibration for defending large language models against jailbreak attacks](https://mlsanity.com/paper/2501.10639/) (2025)
- [EasyEdit2: An Easy-to-use Steering Framework for Editing Large Language Models](https://mlsanity.com/paper/2504.15133/) (2025)
- [Fine-Grained Activation Steering: Steering Less, Achieving More](https://mlsanity.com/paper/2602.04428/) (2026)
- [ReCoVeR the Target Language: Language Steering without Sacrificing Task Performance](https://mlsanity.com/paper/2509.14814/) (2025)
- [What Can We Actually Steer? A Multi-Behavior Study of Activation Control](https://mlsanity.com/paper/2511.18284/) (2025)
- [Enhancing Instruction Following of LLMs via Activation Steering with Dynamic Rejection](https://mlsanity.com/paper/2603.06745/) (2026)
- [Learning to Ponder: Adaptive Reasoning in Latent Space](https://mlsanity.com/paper/2509.24238/) (2025)
- [Steer Like the LLM: Activation Steering that Mimics Prompting](https://mlsanity.com/paper/2605.03907/) (2026)
- [The Hidden Signal of Verifier Strictness: Controlling and Improving Step-Wise Verification via Selective Latent Steering](https://mlsanity.com/paper/2605.20745/) (2026)
- [ExpertSteer: Intervening in LLMs through Expert Knowledge](https://mlsanity.com/paper/2505.12313/) (2025)
- [Depth-Wise Activation Steering for Honest Language Models](https://mlsanity.com/paper/2512.07667/) (2025)
- [The Effectiveness of Style Vectors for Steering Large Language Models: A Human Evaluation](https://mlsanity.com/paper/2601.21505/) (2026)
- [ILRR: Inference-Time Steering Method for Masked Diffusion Language Models](https://mlsanity.com/paper/2601.21647/) (2026)
- [Bias Beyond Borders: Political Ideology Evaluation and Steering in Multilingual LLMs](https://mlsanity.com/paper/2601.23001/) (2026)
- [Activation Steering for Aligned Open-ended Generation without Sacrificing Coherence](https://mlsanity.com/paper/2604.08169/) (2026)
- [Self-Improving Model Steering](https://mlsanity.com/paper/2507.08967/) (2025)
- [Generalization or Memorization: Dynamic Decoding for Mode Steering](https://mlsanity.com/paper/2510.22099/) (2025)
- [Don't Lose Focus: Activation Steering via Key-Orthogonal Projections](https://mlsanity.com/paper/2605.06342/) (2026)
- [Where Steering Signals Come From: Activation Source Selection in Activation Steering](https://mlsanity.com/paper/2607.25270/) (2026)
- [Detecting and Pruning Prominent but Detrimental Neurons in Large Language Models](https://mlsanity.com/paper/2507.09185/) (2025)
- [Exploring Diverse Generation Paths via Inference-time Stiefel Activation Steering](https://mlsanity.com/paper/2601.22010/) (2026)
- [Bifrost: Steering Strategic Trajectories to Bridge Contextual Gaps for Self-Improving Agents](https://mlsanity.com/paper/2602.05810/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have achieved remarkable performance across many tasks, yet aligning them with desired behaviors remains challenging. Activation intervention has emerged as an effective and economical method to modify the behavior of LLMs. Despite considerable interest in this area, current intervention methods exclusively employ a fixed steering vector to modify model activations, lacking adaptability to diverse input semantics. To address this limitation, we propose Semantics-Adaptive Dynamic Intervention (SADI), a novel method that constructs a dynamic steering vector to intervene model activations at inference time. More specifically, SADI utilizes activation differences in contrastive pairs to precisely identify critical elements of an LLM (i.e., attention heads, hidden states, and neurons) for targeted intervention. During inference, SADI dynamically steers model behavior by scaling element-wise activations based on the directions of input semantics. Experimental results show that SADI outperforms established baselines by substantial margins, improving task performance without training. SADI's cost-effectiveness and generalizability across various LLM backbones and tasks highlight its potential as a versatile alignment technique.