mlsanityData to 7 Oct 2026

Activation Scaling for Steering and Interpreting Language Models

arXiv 2410.04962 ↗First posted 7 Oct 2024cs.CL cs.AIcited by 33

Not on the map. Why?

Abstract (arXiv)

Given the prompt "Rome is in", can we steer a language model to flip its prediction of an incorrect token "France" to a correct token "Italy" by only multiplying a few relevant activation vectors with scalars? We argue that successfully intervening on a model is a prerequisite for interpreting its internal workings. Concretely, we establish a three-term objective: a successful intervention should flip the correct with the wrong token and vice versa (effectiveness), and leave other tokens unaffected (faithfulness), all while being sparse (minimality). Using gradient-based optimization, this objective lets us learn (and later evaluate) a specific kind of efficient and interpretable intervention: activation scaling only modifies the signed magnitude of activation vectors to strengthen, weaken, or reverse the steering directions already encoded in the model. On synthetic tasks, this intervention performs comparably with steering vectors in terms of effectiveness and faithfulness, but is much more minimal allowing us to pinpoint interpretable model components. We evaluate activation scaling from different angles, compare performance on different datasets, and make activation scalars a learnable function of the activation vectors themselves to generalize to varying-length prompts.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Builds on 36 of its 48 references that are in the map. First posted on arXiv 7 Oct 2024.

Builds on

36 of 48 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2016Layer Normalizationcited by 5,579
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  6. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  7. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  8. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  9. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  10. 2020On Layer Normalization in the Transformer Architecturecited by 906
  11. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  12. 2023Steering Language Models With Activation Engineeringcited by 824
24 more
  1. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  2. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 571
  3. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  4. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  5. 2021Causal Abstractions of Neural Networkscited by 401
  6. 2021Entity-Based Knowledge Conflicts in Question Answeringcited by 349
  7. 2023Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Modelscited by 237
  8. 2023Attribution Patching Outperforms Automated Circuit Discoverycited by 196
  9. 2020Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Biascited by 183
  10. 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothingcited by 147
  11. 2024Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanismscited by 144
  12. 2023Inspecting and Editing Knowledge Representations in Language Modelscited by 140
  13. 2023Circuit Component Reuse Across Tasks in Transformer Language Modelscited by 105
  14. 2024AtP*: An efficient and scalable method for localizing LLM behaviour to componentscited by 100
  15. 2019The emergence of number and syntax units in LSTM language modelscited by 92
  16. 2024Information Flow Routes: Automatically Interpreting Language Models at Scalecited by 92
  17. 2024Cutting Off the Head Ends the Conflict: A Mechanism for Interpreting and Mitigating Knowledge Conflicts in Language Modelscited by 87
  18. 2024Finding Transformer Circuits with Edge Pruningcited by 58
  19. 2024Do language models plan ahead for future tokens?cited by 47
  20. 2021Differentiable Subset Pruning of Transformer Headscited by 46
  21. 2024Competition of Mechanisms: Tracing How Language Models Handle Facts and Counterfactualscited by 40
  22. 2021Sparse Interventions in Language Models with Differentiable Maskingcited by 34
  23. 2024Backward Lens: Projecting Language Model Gradients into the Vocabulary Spacecited by 30
  24. 2023Generalizing Backpropagation for Gradient-Based Interpretabilitycited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 10; 2026: 22.

33 papers.

Sort:

By area:

  1. 2025AlphaSteer: Learning Refusal Steering with Principled Null-Space Constraintcited by 45
  2. 2025Angular Steering: Behavior Control via Rotation in Activation Spacecited by 45
  3. 2024Controllable Context Sensitivity and the Knob Behind Itcited by 22
  4. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 22
  5. 2026Sycophancy Hides Linearly in the Attention Headscited by 21
  6. 2025Representation Engineering for Large-Language Models: Survey and Research Challengescited by 18
  7. 2025Better Estimation of the Kullback--Leibler Divergence Between Language Modelscited by 10
  8. 2025Unboxing the Black Box: Mechanistic Interpretability for Algorithmic Understanding of Neural Networkscited by 6
  9. 2025ReCoVeR the Target Language: Language Steering without Sacrificing Task Performancecited by 5
  10. 2025Depth-Wise Activation Steering for Honest Language Modelscited by 4
  11. 2026GSS: Gated Subspace Steering for Selective Memorization Mitigation in LLMscited by 2
  12. 2026When is Your LLM Steerable?cited by 2
21 more
  1. 2026Where Steering Signals Come From: Activation Source Selection in Activation Steeringcited by 2
  2. 2025Flash Interpretability: Decoding Specialised Feature Neurons in Large Language Models with the LM-Headcited by 1
  3. 2025Fusion Steering: Prompt-Specific Activation Controlcited by 1
  4. 2025ASGuard: Activation-Scaling Guard to Mitigate Targeted Jailbreaking Attackcited by 1
  5. 2026ShallowJail: Steering Jailbreaks against Large Language Modelscited by 1
  6. 2026Discovering Universal Activation Directions for PII Leakage in Language Modelscited by 1
  7. 2026The Struggle Between Continuation and Refusal: A Mechanistic Analysis of the Continuation-Triggered Jailbreak in LLMscited by 1
  8. 2026FaithSteer-BENCH: A Deployment-Aligned Stress-Testing Benchmark for Inference-Time Steeringcited by 1
  9. 2026Memory Inception: Latent-Space KV Cache Manipulation for Steering LLMscited by 1
  10. 2026Representation-Aware Unlearning via Activation Signatures: From Suppression to Entity-Signature Erasurecited by 0
  11. 2026Asking Forever: Universal Activations Behind Turn Amplification in Conversational LLMscited by 0
  12. 2026The Master Key Hypothesis: Unlocking Cross-Model Capability Transfer via Linear Subspace Alignmentcited by 0
  13. 2026Selective Neuron Amplification in Transformer Language Modelscited by 0
  14. 2026Harnessing Textual Refusal Directions for Multimodal Safetycited by 0
  15. 2026Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Modelscited by 0
  16. 2026Interpretable Representation via LLM-Driven Generative Disentanglement for Local-Life Service Recommendationcited by 0
  17. 2026PCA-guided Activation Scaling for Monotonic Bidirectional Control over LLM Sycophancycited by 0
  18. 2026RARE: Decoupling Representation Steering from Expert Routing in Mixture-of-Experts Language Modelscited by 0
  19. 2026Replay-Gated Neural Execution: Decoupling Persistent Behavioral Specifications from Neural Realizations in Frozen Language Modelscited by 0
  20. 2026Causal Routing for Unlearningcited by 0
  21. 2026Capturing In-Context Learning Dynamics with Task Operatorscited by 0