# Activation Scaling for Steering and Interpreting Language Models

arXiv: https://arxiv.org/abs/2410.04962 · HTML: https://mlsanity.com/paper/2410.04962/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 48 references that are in the map.
- First posted on arXiv 7 Oct 2024.

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](https://mlsanity.com/paper/2211.00593/) (2022)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [Steering Language Models With Activation Engineering](https://mlsanity.com/paper/2308.10248/) (2023)
- [Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned](https://mlsanity.com/paper/1905.09418/) (2019)
- [Towards Automated Circuit Discovery for Mechanistic Interpretability](https://mlsanity.com/paper/2304.14997/) (2023)
- [Learning Sparse Neural Networks through L0 Regularization](https://mlsanity.com/paper/1712.01312/) (2017)
- [Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space](https://mlsanity.com/paper/2203.14680/) (2022)
- [Dissecting Recall of Factual Associations in Auto-Regressive Language Models](https://mlsanity.com/paper/2304.14767/) (2023)
- [Causal Abstractions of Neural Networks](https://mlsanity.com/paper/2106.02997/) (2021)
- [Entity-Based Knowledge Conflicts in Question Answering](https://mlsanity.com/paper/2109.05052/) (2021)
- [Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models](https://mlsanity.com/paper/2301.04213/) (2023)
- [Attribution Patching Outperforms Automated Circuit Discovery](https://mlsanity.com/paper/2310.10348/) (2023)
- [Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias](https://mlsanity.com/paper/2004.12265/) (2020)
- [Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing](https://mlsanity.com/paper/2306.12929/) (2023)
- [Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanisms](https://mlsanity.com/paper/2403.17806/) (2024)
- [Inspecting and Editing Knowledge Representations in Language Models](https://mlsanity.com/paper/2304.00740/) (2023)
- [AtP*: An efficient and scalable method for localizing LLM behaviour to components](https://mlsanity.com/paper/2403.00745/) (2024)
- [Circuit Component Reuse Across Tasks in Transformer Language Models](https://mlsanity.com/paper/2310.08744/) (2023)
- [Information Flow Routes: Automatically Interpreting Language Models at Scale](https://mlsanity.com/paper/2403.00824/) (2024)
- [The emergence of number and syntax units in LSTM language models](https://mlsanity.com/paper/1903.07435/) (2019)
- [Cutting Off the Head Ends the Conflict: A Mechanism for Interpreting and Mitigating Knowledge Conflicts in Language Models](https://mlsanity.com/paper/2402.18154/) (2024)
- [Finding Transformer Circuits with Edge Pruning](https://mlsanity.com/paper/2406.16778/) (2024)

## Built on by (most cited first, 30 of 35)

- [Angular Steering: Behavior Control via Rotation in Activation Space](https://mlsanity.com/paper/2510.26243/) (2025)
- [AlphaSteer: Learning Refusal Steering with Principled Null-Space Constraint](https://mlsanity.com/paper/2506.07022/) (2025)
- [Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Models](https://mlsanity.com/paper/2601.14004/) (2026)
- [Controllable Context Sensitivity and the Knob Behind It](https://mlsanity.com/paper/2411.07404/) (2024)
- [Sycophancy Hides Linearly in the Attention Heads](https://mlsanity.com/paper/2601.16644/) (2026)
- [Representation Engineering for Large-Language Models: Survey and Research Challenges](https://mlsanity.com/paper/2502.17601/) (2025)
- [Better Estimation of the Kullback--Leibler Divergence Between Language Models](https://mlsanity.com/paper/2504.10637/) (2025)
- [ReCoVeR the Target Language: Language Steering without Sacrificing Task Performance](https://mlsanity.com/paper/2509.14814/) (2025)
- [Unboxing the Black Box: Mechanistic Interpretability for Algorithmic Understanding of Neural Networks](https://mlsanity.com/paper/2511.19265/) (2025)
- [Depth-Wise Activation Steering for Honest Language Models](https://mlsanity.com/paper/2512.07667/) (2025)
- [When is Your LLM Steerable?](https://mlsanity.com/paper/2606.11599/) (2026)
- [Where Steering Signals Come From: Activation Source Selection in Activation Steering](https://mlsanity.com/paper/2607.25270/) (2026)
- [Mitigating Memorization Where It Happens](https://mlsanity.com/paper/2602.08901/) (2026)
- [FaithSteer-BENCH: A Deployment-Aligned Stress-Testing Benchmark for Inference-Time Steering](https://mlsanity.com/paper/2603.18329/) (2026)
- [Memory Inception: Latent-Space KV Cache Manipulation for Steering LLMs](https://mlsanity.com/paper/2605.06225/) (2026)
- [Flash Interpretability: Decoding Specialised Feature Neurons in Large Language Models with the LM-Head](https://mlsanity.com/paper/2501.02688/) (2025)
- [Fusion Steering: Prompt-Specific Activation Control](https://mlsanity.com/paper/2505.22572/) (2025)
- [ASGuard: Activation-Scaling Guard to Mitigate Targeted Jailbreaking Attack](https://mlsanity.com/paper/2509.25843/) (2025)
- [ShallowJail: Steering Jailbreaks against Large Language Models](https://mlsanity.com/paper/2602.07107/) (2026)
- [Discovering Universal Activation Directions for PII Leakage in Language Models](https://mlsanity.com/paper/2602.16980/) (2026)
- [The Struggle Between Continuation and Refusal: A Mechanistic Analysis of the Continuation-Triggered Jailbreak in LLMs](https://mlsanity.com/paper/2603.08234/) (2026)
- [The Master Key Hypothesis: Unlocking Cross-Model Capability Transfer via Linear Subspace Alignment](https://mlsanity.com/paper/2604.06377/) (2026)
- [Representation-Aware Unlearning via Activation Signatures: From Suppression to Entity-Signature Erasure](https://mlsanity.com/paper/2601.10566/) (2026)
- [Asking Forever: Universal Activations Behind Turn Amplification in Conversational LLMs](https://mlsanity.com/paper/2602.17778/) (2026)
- [Selective Neuron Amplification in Transformer Language Models](https://mlsanity.com/paper/2604.07098/) (2026)
- [Harnessing Textual Refusal Directions for Multimodal Safety](https://mlsanity.com/paper/2606.31876/) (2026)
- [Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models](https://mlsanity.com/paper/2607.26326/) (2026)
- [Interpretable Representation via LLM-Driven Generative Disentanglement for Local-Life Service Recommendation](https://mlsanity.com/paper/2607.27944/) (2026)
- [PCA-guided Activation Scaling for Monotonic Bidirectional Control over LLM Sycophancy](https://mlsanity.com/paper/2608.16650/) (2026)
- [RARE: Decoupling Representation Steering from Expert Routing in Mixture-of-Experts Language Models](https://mlsanity.com/paper/2608.21236/) (2026)

## Abstract (arXiv)

Given the prompt "Rome is in", can we steer a language model to flip its prediction of an incorrect token "France" to a correct token "Italy" by only multiplying a few relevant activation vectors with scalars? We argue that successfully intervening on a model is a prerequisite for interpreting its internal workings. Concretely, we establish a three-term objective: a successful intervention should flip the correct with the wrong token and vice versa (effectiveness), and leave other tokens unaffected (faithfulness), all while being sparse (minimality). Using gradient-based optimization, this objective lets us learn (and later evaluate) a specific kind of efficient and interpretable intervention: activation scaling only modifies the signed magnitude of activation vectors to strengthen, weaken, or reverse the steering directions already encoded in the model. On synthetic tasks, this intervention performs comparably with steering vectors in terms of effectiveness and faithfulness, but is much more minimal allowing us to pinpoint interpretable model components. We evaluate activation scaling from different angles, compare performance on different datasets, and make activation scalars a learnable function of the activation vectors themselves to generalize to varying-length prompts.