mlsanityData to 7 Oct 2026

Post-Routing Arithmetic in Llama-3: Last-Token Result Writing and Rotation-Structured Digit Directions

arXiv 2602.19109 ↗First posted 22 Feb 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

We study three-digit addition in Meta-Llama-3-8B (base) under a one-token readout to characterize how arithmetic answers are finalized after cross-token routing becomes causally irrelevant. Causal residual patching and cumulative attention ablations localize a sharp boundary near layer~17: beyond it, the decoded sum is controlled almost entirely by the last input token and late-layer self-attention is largely dispensable. In this post-routing regime, digit(-sum) direction dictionaries vary with a next-higher-digit context but are well-related by an approximately orthogonal map inside a shared low-rank subspace (low-rank Procrustes alignment). Causal digit editing matches this geometry: naive cross-context transfer fails, while rotating directions through the learned map restores strict counterfactual edits; negative controls do not recover.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 10 of its 16 references that are in the map. First posted on arXiv 22 Feb 2026.

Builds on

10 of 16 references.

References · 10

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  3. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  4. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  5. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 328
  6. 2024How to use and interpret activation patchingcited by 191
  7. 2024Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristicscited by 94
  8. 2024Pre-trained Large Language Models Use Fourier Features to Compute Additioncited by 38
  9. 2023Understanding Addition in Transformerscited by 34
  10. 2025All for One: LLMs Solve Mental Math at the Last Token With Information Transferred From Other Tokenscited by 5

Built on by

Not cited yet.