Necessary, Decodable and Reversible, Yet Not Transferable: A Stress Test for Attention-Head Role Claims
Not on the map. Why?
Abstract (arXiv)
Mechanistic studies often assign a component a role when removing it damages a behavior, its activation linearly encodes task information, and restoring that activation repairs the damage. We test the stronger implication: can the same activation carry the requested computation into another prompt? In the roughly 8 of 15 model-family cells receiving our full matched-control transfer assay, covering three 7-8B instruction-tuned models and 5 single-step computation families, tested attention-head states did not produce clean computation transfer. Same-answer and matched-context controls instead exposed inert or broad effects. Positive controls bound this result rather than eliminating its caveats: the instrument recovers known function vectors when writing into an underspecified prompt, and recovers a compact override-capable residual state in one of three controlled-model seeds. Neither control proves sensitivity to the object carried by real-model attention during prompt override. On two composed forms in Qwen, a strongly decodable operation direction also failed to steer the generated answer under passing gates; the corresponding cross-model test was invalid. These results show that necessity, decodability, same-prompt repair, and cross-prompt transfer are separable evidence. They do not establish that a portable selection state is absent from untested components or settings.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 24 of its 27 references that are in the map. First posted on arXiv 6 Jun 2026.
Builds on
24 of 27 references.
References · 24
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2024Qwen2.5 Technical Reportcited by 4,942
- 2023Mistral 7Bcited by 3,164
- 2022Locating and Editing Factual Associations in GPTcited by 2,691
- 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
- 2022Toy Models of Superpositioncited by 971
- 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
- 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 767
- 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 724
- 2019Designing and Interpreting Probes with Control Taskscited by 640
- 2023In-Context Learning Creates Task Vectorscited by 312
- 2023Function Vectors in Large Language Modelscited by 308
12 more
- 2024How to use and interpret activation patchingcited by 214
- 2023Linearity of Relation Decoding in Transformer Language Modelscited by 200
- 2025Open Problems in Mechanistic Interpretabilitycited by 179
- 2023Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patchingcited by 80
- 2023Copy Suppression: Comprehensively Understanding an Attention Headcited by 76
- 2025Which Attention Heads Matter for In-Context Learning?cited by 61
- 2024The Super Weight in Large Language Modelscited by 52
- 2023Interpretability Illusions in the Generalization of Simplified Modelscited by 25
- 2025Mechanistic Interpretability as Statistical Estimation: A Variance Analysiscited by 15
- 2026Causality neq Invariance: Function and Concept Vectors in LLMscited by 8
- 2025Beyond Components: Singular Vector-Based Interpretability of Transformer Circuitscited by 7
- 2026Make Mechanistic Interpretability Auditable: A Call to Develop Guidelines via Continuous Collaborative Reviewingcited by 2
Built on by
Not cited yet.