Post-Routing Arithmetic in Llama-3: Last-Token Result Writing and Rotation-Structured Digit Directions
Not on the map. Why?
Abstract (arXiv)
We study three-digit addition in Meta-Llama-3-8B (base) under a one-token readout to characterize how arithmetic answers are finalized after cross-token routing becomes causally irrelevant. Causal residual patching and cumulative attention ablations localize a sharp boundary near layer~17: beyond it, the decoded sum is controlled almost entirely by the last input token and late-layer self-attention is largely dispensable. In this post-routing regime, digit(-sum) direction dictionaries vary with a next-higher-digit context but are well-related by an approximately orthogonal map inside a shared low-rank subspace (low-rank Procrustes alignment). Causal digit editing matches this geometry: naive cross-context transfer fails, while rotating directions through the learned map restores strict counterfactual edits; negative controls do not recover.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 10 of its 16 references that are in the map. First posted on arXiv 22 Feb 2026.
Builds on
10 of 16 references.
References · 10
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2022Locating and Editing Factual Associations in GPTcited by 2,609
- 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
- 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
- 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 328
- 2024How to use and interpret activation patchingcited by 191
- 2024Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristicscited by 94
- 2024Pre-trained Large Language Models Use Fourier Features to Compute Additioncited by 38
- 2023Understanding Addition in Transformerscited by 34
- 2025All for One: LLMs Solve Mental Math at the Last Token With Information Transferred From Other Tokenscited by 5
Built on by
Not cited yet.