# Every Ablation Is a Dose: Counterweights and the Semblance of Self-Repair

arXiv: https://arxiv.org/abs/2610.02173 · HTML: https://mlsanity.com/paper/2610.02173/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 46 references that are in the map.
- First posted on arXiv 1 Oct 2026.

## Builds on (most cited first, 23 of 23)

- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [Representation Engineering: A Top-Down Approach to AI Transparency](https://mlsanity.com/paper/2310.01405/) (2023)
- [Steering Language Models With Activation Engineering](https://mlsanity.com/paper/2308.10248/) (2023)
- [Towards Automated Circuit Discovery for Mechanistic Interpretability](https://mlsanity.com/paper/2304.14997/) (2023)
- [Eliciting Latent Predictions from Transformers with the Tuned Lens](https://mlsanity.com/paper/2303.08112/) (2023)
- [Learning Sparse Neural Networks through L0 Regularization](https://mlsanity.com/paper/1712.01312/) (2017)
- [Causal Abstractions of Neural Networks](https://mlsanity.com/paper/2106.02997/) (2021)
- [Towards Best Practices of Activation Patching in Language Models: Metrics and Methods](https://mlsanity.com/paper/2309.16042/) (2023)
- [How to use and interpret activation patching](https://mlsanity.com/paper/2404.15255/) (2024)
- [Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias](https://mlsanity.com/paper/2004.12265/) (2020)
- [The Hydra Effect: Emergent Self-repair in Language Model Computations](https://mlsanity.com/paper/2307.15771/) (2023)
- [An Interpretability Illusion for BERT](https://mlsanity.com/paper/2104.07143/) (2021)
- [Explorations of Self-Repair in Language Models](https://mlsanity.com/paper/2402.15390/) (2024)
- [Optimal ablation for interpretability](https://mlsanity.com/paper/2409.09951/) (2024)
- [Transformer Circuit Faithfulness Metrics are not Robust](https://mlsanity.com/paper/2407.08734/) (2024)
- [A Reply to Makelov et al. (2023)'s "Interpretability Illusion" Arguments](https://mlsanity.com/paper/2401.12631/) (2024)
- [The Curse of Multiple Mediators: Hidden Interaction Effects in Activation Patching](https://mlsanity.com/paper/2606.27510/) (2026)
- [Rethinking Circuit Completeness in Language Models: AND, OR, and ADDER Gates](https://mlsanity.com/paper/2505.10039/) (2025)
- [Conditional Co-Ablation: Recovering Self-Repair Backups in Transformer Circuits](https://mlsanity.com/paper/2607.01940/) (2026)
- [Beyond Components: Singular Vector-Based Interpretability of Transformer Circuits](https://mlsanity.com/paper/2511.20273/) (2025)
- [Function-Vector Heads Are Two Populations: Writers and Cancellers in In-Context Learning](https://mlsanity.com/paper/2606.07560/) (2026)
- [Neural Natural Language Inference Models Partially Embed Theories of Lexical Entailment and Negation](https://mlsanity.com/paper/2004.14623/) (2020)
- [Through the Looking Glass: Directly Reading and Writing Transformers](https://mlsanity.com/paper/2609.10210/) (2026)

## Abstract (arXiv)

Ablate a component of a language model, and other components often appear to adjust and compensate. This phenomenon, termed self-repair, has been observed repeatedly, but its mechanism remains unclear. The most systematic study to date concluded that self-repair is noisy and unlikely to have a single explanation. We argue that it has one: a gain already present before any ablation. Any intervention on a causally important component can be viewed as a point on a coordinate axis λ, the signed strength of a counterfactual contrast. Hence, conventional ablation methods are uncalibrated points on this axis. We show that the causal repair response for a fine-grained unit r is governed by an affine law, Er(λ)=ownr+γrλ. The slope γr is a fixed coefficient that consistently influences the model, with or without ablation, and its sign determines whether the unit counteracts or reinforces the removed signal. On a factual-verdict task across four models from distinct families (Gemma, Qwen, LLaMA, and Mistral), we identify components including MLP neurons, OV neurons, and singular directions that follow this affine law, 68 of 81 downstream directions in all. Moreover, we can anticipate the magnitude of γr from the fixed weights. On the IOI circuit of GPT-2 Small, seven of the ten heads the intervention can reach follow the law, and all seven are counterweights. From this perspective, what may appear as self-repair is a counterweight performing its usual operation when the contrastive signal emerges at the core.