# Understanding Addition and Subtraction in Transformers

arXiv: https://arxiv.org/abs/2402.02619 · HTML: https://mlsanity.com/paper/2402.02619/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 21 references that are in the map.
- First posted on arXiv 4 Feb 2024.

## Builds on (most cited first, 9 of 9)

- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023)
- [Towards Automated Circuit Discovery for Mechanistic Interpretability](https://mlsanity.com/paper/2304.14997/) (2023)
- [The Quantization Model of Neural Scaling](https://mlsanity.com/paper/2303.13506/) (2023)
- [Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Tasks](https://mlsanity.com/paper/2305.14201/) (2023)
- [GPT Can Solve Mathematical Problems Without a Calculator](https://mlsanity.com/paper/2309.03241/) (2023)
- [Memorization Capacity of Multi-Head Attention in Transformers](https://mlsanity.com/paper/2306.02010/) (2023)
- [Understanding Addition in Transformers](https://mlsanity.com/paper/2310.13121/) (2023)

## Built on by (most cited first, 8 of 8)

- [Mechanistic Interpretability for AI Safety -- A Review](https://mlsanity.com/paper/2404.14082/) (2024)
- [Arithmetic in the Wild: Llama uses Base-10 Addition to Reason About Cyclic Concepts](https://mlsanity.com/paper/2605.01148/) (2026)
- [Unveiling Language Competence Neurons: A Psycholinguistic Approach to Model Interpretability](https://mlsanity.com/paper/2409.15827/) (2024)
- [Survival of the Fittest Representation: A Case Study with Modular Addition](https://mlsanity.com/paper/2405.17420/) (2024)
- [Dynamic Latent Routing](https://mlsanity.com/paper/2605.14323/) (2026)
- [The Shape of Addition: Geometric Structures of Arithmetic in Large Language Models](https://mlsanity.com/paper/2606.03645/) (2026)
- [Language Models Compare Quantities Using Number-specific and Unit-specific Heuristics](https://mlsanity.com/paper/2606.03982/) (2026)
- [Distance generalization in transformers: why bother with positional encoding?](https://mlsanity.com/paper/2609.11913/) (2026)

## Abstract (arXiv)

We use integer addition and subtraction as a controlled, exactly-solvable testbed for what can be said with confidence about the algorithm a low-loss transformer implements - logically and mechanically. We train small transformers (2-3 layers) from scratch, find the edge cases they fail (long carry and borrow cascades), and enrich the training data with them; most resulting models reach >99.999% accuracy on 5-15 digit problems in under an hour, solving cascades such as 555555555+444444448=+1000000003. Interpretability analysis of these accurate models - not an a-priori guess - surfaces the structure from which we derive novel, sound, exact left-to-right algorithms for both operations, matching the causal ordering transformers must use. We then put these algorithms back to the models: across 46 models varying in seed, depth, digit-length, and operation, the accurate ones share one circuit skeleton, and in representative models deeper causal analysis (activation and edge path-patching, with controls) confirms the multi-digit resolved-carry sub-task that earlier analyses left open - while ruling out several candidate mechanisms. The skeleton recurs across model sizes and operations, and mixed-model nodes are polysemantic across addition and subtraction whether initialized from an addition model or trained from scratch. We release the models, analysis, and a reproducible interpretability toolkit.