mlsanityData to 7 Oct 2026

Semantics-Adaptive Activation Intervention for LLMs via Dynamic Steering Vectors

arXiv 2410.12299 ↗First posted 16 Oct 2024cs.CLcited by 61

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have achieved remarkable performance across many tasks, yet aligning them with desired behaviors remains challenging. Activation intervention has emerged as an effective and economical method to modify the behavior of LLMs. Despite considerable interest in this area, current intervention methods exclusively employ a fixed steering vector to modify model activations, lacking adaptability to diverse input semantics. To address this limitation, we propose Semantics-Adaptive Dynamic Intervention (SADI), a novel method that constructs a dynamic steering vector to intervene model activations at inference time. More specifically, SADI utilizes activation differences in contrastive pairs to precisely identify critical elements of an LLM (i.e., attention heads, hidden states, and neurons) for targeted intervention. During inference, SADI dynamically steers model behavior by scaling element-wise activations based on the directions of input semantics. Experimental results show that SADI outperforms established baselines by substantial margins, improving task performance without training. SADI's cost-effectiveness and generalizability across various LLM backbones and tasks highlight its potential as a versatile alignment technique.

Key facts. Cited by 61 papers in the map (data through 7 Oct 2026). Builds on 29 of its 39 references that are in the map. First posted on arXiv 16 Oct 2024.

Builds on

29 of 39 references.

References · 29

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2023Mistral 7Bcited by 3,065
  7. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  8. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  9. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  10. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  11. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  12. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
17 more
  1. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  2. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,009
  3. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  4. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  5. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
  6. 2023The Falcon Series of Open Language Modelscited by 526
  7. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 419
  8. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 413
  9. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 330
  10. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 240
  11. 2022Extracting Latent Steering Vectors from Pretrained Language Modelscited by 217
  12. 2024How do Large Language Models Handle Multilingualism?cited by 148
  13. 2023Inspecting and Editing Knowledge Representations in Language Modelscited by 140
  14. 2023Truth Forest: Toward Multi-Scale Truthfulness in Large Language Models through Intervention without Tuningcited by 45
  15. 2024Sharing Matters: Analysing Neurons Across Languages and Tasks in LLMscited by 33
  16. 2023Retrieval-augmented Multilingual Knowledge Editingcited by 13
  17. 2023Interpreting and Exploiting Functional Specialization in Multi-Head Attention under Multi-task Learningcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 17; 2026: 44.

61 papers.

Sort:

By area:

  1. 2025Angular Steering: Behavior Control via Rotation in Activation Spacecited by 45
  2. 2025Representation Engineering for Large-Language Models: Survey and Research Challengescited by 18
  3. 2026Spherical Steering: Geometry-Aware Activation Rotation for Language Modelscited by 17
  4. 2025LLM-CAS: Dynamic Neuron Perturbation for Real-Time Hallucination Correctioncited by 15
  5. 2026ODESteer: A Unified ODE-Based Steering Framework for LLM Alignmentcited by 15
  6. 2025KV Cache Steering for Controlling Frozen LLMscited by 8
  7. 2025Latent-space adversarial training with post-aware calibration for defending large language models against jailbreak attackscited by 7
  8. 2025EasyEdit2: An Easy-to-use Steering Framework for Editing Large Language Modelscited by 7
  9. 2025Steering When Necessary: Flexible Steering Large Language Models with Backtrackingcited by 7
  10. 2025What Can We Actually Steer? A Multi-Behavior Study of Activation Controlcited by 6
  11. 2026Fine-Grained Activation Steering: Steering Less, Achieving Morecited by 6
  12. 2026Dynamic Multimodal Activation Steering for Hallucination Mitigation in Large Vision-Language Modelscited by 6
38 more
  1. 2025ReCoVeR the Target Language: Language Steering without Sacrificing Task Performancecited by 5
  2. 2026The Hidden Signal of Verifier Strictness: Controlling and Improving Step-Wise Verification via Selective Latent Steeringcited by 5
  3. 2025ExpertSteer: Intervening in LLMs through Expert Knowledgecited by 4
  4. 2025Learning to Ponder: Adaptive Reasoning in Latent Spacecited by 4
  5. 2025Depth-Wise Activation Steering for Honest Language Modelscited by 4
  6. 2026Bias Beyond Borders: Political Ideology Evaluation and Steering in Multilingual LLMscited by 4
  7. 2026Enhancing Instruction Following of LLMs via Activation Steering with Dynamic Rejectioncited by 4
  8. 2025Self-Improving Model Steeringcited by 3
  9. 2025Generalization or Memorization: Dynamic Decoding for Mode Steeringcited by 3
  10. 2026The Effectiveness of Style Vectors for Steering Large Language Models: A Human Evaluationcited by 3
  11. 2026ILRR: Inference-Time Steering Method for Masked Diffusion Language Modelscited by 3
  12. 2025Detecting and Pruning Prominent but Detrimental Neurons in Large Language Modelscited by 2
  13. 2026Exploring Diverse Generation Paths via Inference-time Stiefel Activation Steeringcited by 2
  14. 2026Bifrost: Steering Strategic Trajectories to Bridge Contextual Gaps for Self-Improving Agentscited by 2
  15. 2026Activation Steering for Aligned Open-ended Generation without Sacrificing Coherencecited by 2
  16. 2026Neuron-Anchored Rule Extraction for Large Language Models via Contrastive Hierarchical Ablationcited by 2
  17. 2026Don't Lose Focus: Activation Steering via Key-Orthogonal Projectionscited by 2
  18. 2026Inference-Time Machine Unlearning via Gated Activation Redirectioncited by 2
  19. 2026Where Steering Signals Come From: Activation Source Selection in Activation Steeringcited by 2
  20. 2025Head-Specific Intervention Can Induce Misaligned AI Coordination in Large Language Modelscited by 1
  21. 2025The Adaptation Paradox: Agency vs. Mimicry in Companion Chatbotscited by 1
  22. 2026RankGuide: Tensor-Rank-Guided Routing and Steering for Efficient Reasoningcited by 1
  23. 2026Steer Like the LLM: Activation Steering that Mimics Promptingcited by 1
  24. 2026Chain of Risk: Safety Failures in Large Reasoning Models and Mitigation via Adaptive Multi-Principle Steeringcited by 1
  25. 2026Hallucination as an Anomaly: Dynamic Intervention via Probabilistic Circuitscited by 1
  26. 2026Resist and Update: Counterfactual Report Coordinates for Incentive-Compatible LLMscited by 1
  27. 2026Spillover-Aware Multi-Value Steering for Pluralistic LLM Alignmentcited by 1
  28. 2026Watch the Model Think: On-Policy Extraction of Activation Steering Vectorscited by 0
  29. 2026Continuous Interpretive Steering for Scalar Diversitycited by 0
  30. 2026Causal Probing for Internal Visual Representations in Multimodal Large Language Modelscited by 0
  31. 2026Mitigating Mask Prior Drift and Positional Attention Collapse in Large Diffusion Vision-Language Modelscited by 0
  32. 2026Residual Paving: Diagnosing the Routing Bottleneck in Selective Refusal Editingcited by 0
  33. 2026Steered Generation via Gradient-Based Optimization on Sparse Query Featurescited by 0
  34. 2026Mitigating Object Hallucinations in Vision-Language Models through Region-Aware Attention Recalibrationcited by 0
  35. 2026Activation Steering for Synthetic Data Generation: The Role of Diversity in Downstream Safety Detectioncited by 0
  36. 2026Learning to Refine Hidden States for Reliable LLM Reasoningcited by 0
  37. 2026Decodable Is Not Grounded: A Vision-Ablation Arbiter for VLM Spatial Reasoningcited by 0
  38. 2026Subliminal Clocks: Latent Time Modelling in Diffusion Language Modelscited by 0

50 most cited of 61 All 61 (JSON)