# Weight Updates as Activation Shifts: A Principled Framework for Steering

arXiv: https://arxiv.org/abs/2603.00425 · HTML: https://mlsanity.com/paper/2603.00425/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Activation steering methods, one of 13 directions in Mechanistic interpretability.
- Builds on 24 of its 33 references that are in the map.
- First posted on arXiv 28 Feb 2026.

## Placed in

- Mechanistic interpretability / [Activation steering methods](https://mlsanity.com/mechinterp/activation-steering-methods/)

## Builds on (most cited first, 24 of 24)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [DoRA: Weight-Decomposed Low-Rank Adaptation](https://mlsanity.com/paper/2402.09353/) (2024)
- [Transformers learn in-context by gradient descent](https://mlsanity.com/paper/2212.07677/) (2022)
- [Language Modeling with Gated Convolutional Networks](https://mlsanity.com/paper/1612.08083/) (2016)
- [Long Range Arena: A Benchmark for Efficient Transformers](https://mlsanity.com/paper/2011.04006/) (2020)
- [Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problems](https://mlsanity.com/paper/1705.04146/) (2017)
- [Trained Transformers Learn Linear Models In-Context](https://mlsanity.com/paper/2306.09927/) (2023)
- [Function Vectors in Large Language Models](https://mlsanity.com/paper/2310.15213/) (2023)
- [ReFT: Representation Finetuning for Language Models](https://mlsanity.com/paper/2404.03592/) (2024)
- [ListOps: A Diagnostic Dataset for Latent Tree Learning](https://mlsanity.com/paper/1804.06028/) (2018)
- [LoFiT: Localized Fine-tuning on LLM Representations](https://mlsanity.com/paper/2406.01563/) (2024)
- [Learning without training: The implicit dynamics of in-context learning](https://mlsanity.com/paper/2507.16003/) (2025)
- [Belief Dynamics Reveal the Dual Nature of In-Context Learning and Activation Steering](https://mlsanity.com/paper/2511.00617/) (2025)
- [Joint Localization and Activation Editing for Low-Resource Fine-Tuning](https://mlsanity.com/paper/2502.01179/) (2025)
- [Is Free Self-Alignment Possible?](https://mlsanity.com/paper/2406.03642/) (2024)

## Built on by (most cited first, 3 of 3)

- [How Much Do Circuits Tell Us? Measuring the Consistency and Specificity of Language Model Circuits](https://mlsanity.com/paper/2605.08348/) (2026)
- [Activation-Space Order-Swap Geometry: A Site-Asymmetry Audit](https://mlsanity.com/paper/2608.25315/) (2026)
- [Learning to Decide, Not to Reason: Parameter-Efficient Decision Operators via Low-Rank Activation Steering](https://mlsanity.com/paper/2610.06950/) (2026)

## Abstract (arXiv)

Activation steering promises to be an extremely parameter-efficient form of adaptation, but its effectiveness depends on critical design choices -- such as intervention location and parameterization -- that currently rely on empirical heuristics rather than a principled foundation. We establish a first-order equivalence between activation-space interventions and weight-space updates, deriving the conditions under which activation steering can replicate fine-tuning behavior. This equivalence yields a principled framework for steering design and identifies the post-block output as a theoretically-backed and highly expressive intervention site. We further explain why certain intervention locations outperform others and show that weight updates and activation updates play distinct, complementary functional roles. This analysis motivates a new approach -- joint adaptation -- that trains in both spaces simultaneously. Our post-block steering achieves accuracy within 0.2%-0.9% of full-parameter tuning, on average across tasks and models, while training only 0.04% of model parameters. It consistently outperforms prior activation steering methods such as ReFT and PEFT approaches including LoRA, while using significantly fewer parameters. Finally, we show that joint adaptation often surpasses the performance ceilings of weight and activation updates in isolation, introducing a new paradigm for efficient model adaptation.