# Understanding Addition in Transformers

arXiv: https://arxiv.org/abs/2310.13121 · HTML: https://mlsanity.com/paper/2310.13121/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 21 references that are in the map.
- First posted on arXiv 19 Oct 2023.

## Builds on (most cited first, 12 of 12)

- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023)
- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [Towards Automated Circuit Discovery for Mechanistic Interpretability](https://mlsanity.com/paper/2304.14997/) (2023)
- [Network Dissection: Quantifying Interpretability of Deep Visual Representations](https://mlsanity.com/paper/1704.05796/) (2017)
- [Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients](https://mlsanity.com/paper/1912.04871/) (2019)
- [A Multiscale Visualization of Attention in the Transformer Model](https://mlsanity.com/paper/1906.05714/) (2019)
- [Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks](https://mlsanity.com/paper/2207.13243/) (2022)
- [Interpreting Learned Feedback Patterns in Large Language Models](https://mlsanity.com/paper/2310.08164/) (2023)
- [Creative AI Through Evolutionary Computation](https://mlsanity.com/paper/1901.03775/) (2019)

## Built on by (most cited first, 30 of 34)

- [Mechanistic Interpretability for AI Safety -- A Review](https://mlsanity.com/paper/2404.14082/) (2024)
- [Talking Heads: Understanding Inter-layer Communication in Transformer Language Models](https://mlsanity.com/paper/2406.09519/) (2024)
- [Self-Improving Transformers Overcome Easy-to-Hard and Length Generalization Challenges](https://mlsanity.com/paper/2502.01612/) (2025)
- [Hypothesis Testing the Circuit Hypothesis in LLMs](https://mlsanity.com/paper/2410.13032/) (2024)
- [Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Models](https://mlsanity.com/paper/2601.14004/) (2026)
- [Position Coupling: Improving Length Generalization of Arithmetic Transformers Using Task Structure](https://mlsanity.com/paper/2405.20671/) (2024)
- [Opening the AI black box: program synthesis via mechanistic interpretability](https://mlsanity.com/paper/2402.05110/) (2024)
- [Algorithmic Capabilities of Random Transformers](https://mlsanity.com/paper/2410.04368/) (2024)
- [Transformers Learn Low Sensitivity Functions: Investigations and Implications](https://mlsanity.com/paper/2403.06925/) (2024)
- [Extrapolation by Association: Length Generalization Transfer in Transformers](https://mlsanity.com/paper/2506.09251/) (2025)
- [Explicitly Encoding Structural Symmetry is Key to Length Generalization in Arithmetic Tasks](https://mlsanity.com/paper/2406.01895/) (2024)
- [What Does It Mean to Be a Transformer? Insights from a Theoretical Hessian Analysis](https://mlsanity.com/paper/2410.10986/) (2024)
- [Understanding Addition and Subtraction in Transformers](https://mlsanity.com/paper/2402.02619/) (2024)
- [InversionView: A General-Purpose Method for Reading Information from Neural Activations](https://mlsanity.com/paper/2405.17653/) (2024)
- [Mechanism and Emergence of Stacked Attention Heads in Multi-Layer Transformers](https://mlsanity.com/paper/2411.12118/) (2024)
- [Counting in Small Transformers: The Delicate Interplay between Attention and Feed-Forward Layers](https://mlsanity.com/paper/2407.11542/) (2024)
- [Decoupling Positional and Symbolic Attention Behavior in Transformers](https://mlsanity.com/paper/2511.11579/) (2025)
- [Clustering and Alignment: Understanding the Training Dynamics in Modular Addition](https://mlsanity.com/paper/2408.09414/) (2024)
- [Validating Mechanistic Interpretations: An Axiomatic Approach](https://mlsanity.com/paper/2407.13594/) (2024)
- [Addition in Four Movements: Mapping Layer-wise Information Trajectories in LLMs](https://mlsanity.com/paper/2506.07824/) (2025)
- [Path Channels and Plan Extension Kernels: a Mechanistic Description of Planning in a Sokoban RNN](https://mlsanity.com/paper/2506.10138/) (2025)
- [Learning the symmetric group: large from small](https://mlsanity.com/paper/2502.12717/) (2025)
- [Efficient Knowledge Probing of Large Language Models by Adapting Pre-trained Embeddings](https://mlsanity.com/paper/2508.06030/) (2025)
- [The Long Delay to Arithmetic Generalization: When Learned Representations Outrun Behavior](https://mlsanity.com/paper/2604.13082/) (2026)
- [Understanding In-Context Learning for Nonlinear Regression with Transformers: Attention as Featurizer](https://mlsanity.com/paper/2605.05176/) (2026)
- [From Prediction to Understanding: Will AI Foundation Models Transform Brain Science?](https://mlsanity.com/paper/2509.17280/) (2025)
- [Arithmetic OOD Failure Unfolds in Stages in Minimal GPTs](https://mlsanity.com/paper/2603.26828/) (2026)
- [Learning Modular Exponentiation with Transformers](https://mlsanity.com/paper/2506.23679/) (2025)
- [Breaking the Data Barrier in Learning Symbolic Computation: A Case Study on Variable Ordering Suggestion for Cylindrical Algebraic Decomposition](https://mlsanity.com/paper/2601.13731/) (2026)
- [The Anxiety of Influence: Bloom Filters in Transformer Attention Heads](https://mlsanity.com/paper/2602.17526/) (2026)

## Abstract (arXiv)

Understanding the inner workings of machine learning models like Transformers is vital for their safe and ethical use. This paper provides a comprehensive analysis of a one-layer Transformer model trained to perform n-digit integer addition. Our findings suggest that the model dissects the task into parallel streams dedicated to individual digits, employing varied algorithms tailored to different positions within the digits. Furthermore, we identify a rare scenario characterized by high loss, which we explain. By thoroughly elucidating the model's algorithm, we provide new insights into its functioning. These findings are validated through rigorous testing and mathematical modeling, thereby contributing to the broader fields of model understanding and interpretability. Our approach opens the door for analyzing more complex tasks and multi-layer Transformer models.