# Modular Arithmetic: Language Models Solve Math Digit by Digit

arXiv: https://arxiv.org/abs/2508.02513 · HTML: https://mlsanity.com/paper/2508.02513/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 28 references that are in the map.
- First posted on arXiv 4 Aug 2025.

## Builds on (most cited first, 14 of 14)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023)
- [Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristics](https://mlsanity.com/paper/2410.21272/) (2024)
- [Successor Heads: Recurring, Interpretable Attention Heads In The Wild](https://mlsanity.com/paper/2312.09230/) (2023)
- [Language Models Use Trigonometry to Do Addition](https://mlsanity.com/paper/2502.00873/) (2025)
- [pyvene: A Library for Understanding and Improving PyTorch Models via Interventions](https://mlsanity.com/paper/2403.07809/) (2024)
- [Interpreting and Improving Large Language Models in Arithmetic Calculation](https://mlsanity.com/paper/2409.01659/) (2024)
- [Pre-trained Large Language Models Use Fourier Features to Compute Addition](https://mlsanity.com/paper/2406.03445/) (2024)
- [Language Models Encode Numbers Using Digit Representations in Base 10](https://mlsanity.com/paper/2410.11781/) (2024)
- [Language Models Encode the Value of Numbers Linearly](https://mlsanity.com/paper/2401.03735/) (2024)
- [What is a Number, That a Large Language Model May Know It?](https://mlsanity.com/paper/2502.01540/) (2025)
- [The Lookahead Limitation: Why Multi-Operand Addition is Hard for LLMs](https://mlsanity.com/paper/2502.19981/) (2025)
- [Language Models are Symbolic Learners in Arithmetic](https://mlsanity.com/paper/2410.15580/) (2024)

## Built on by (most cited first, 6 of 6)

- [Language Model Circuits Are Sparse in the Neuron Basis](https://mlsanity.com/paper/2601.22594/) (2026)
- [AI-rithmetic](https://mlsanity.com/paper/2602.10416/) (2026)
- [Are Arithmetic Heuristic Neurons Form-Invariant? A Mechanistic Analysis of Symbols, Text, and Code in LLMs](https://mlsanity.com/paper/2607.16693/) (2026)
- [Automated Attention Pattern Discovery at Scale in Large Language Models](https://mlsanity.com/paper/2604.03764/) (2026)
- [Disentangling Mathematical Reasoning in LLMs: A Methodological Investigation of Internal Mechanisms](https://mlsanity.com/paper/2604.15842/) (2026)
- [When Tokenization is Secretly Output Supervision](https://mlsanity.com/paper/2609.01386/) (2026)

## Abstract (arXiv)

While recent work has begun to uncover the internal strategies that Large Language Models (LLMs) employ for simple arithmetic tasks, a unified understanding of their underlying mechanisms is still lacking. We extend recent findings showing that LLMs represent numbers in a digit-wise manner and present evidence for the existence of digit-position-specific circuits that LLMs use to perform simple arithmetic tasks, i.e. modular subgroups of MLP neurons that operate independently on different digit positions (units, tens, hundreds). Notably, such circuits exist independently of model size and of tokenization strategy, i.e. both for models that encode longer numbers digit-by-digit and as one token. Using Feature Importance and Causal Interventions, we identify and validate the digit-position-specific circuits, revealing a compositional and interpretable structure underlying the solving of arithmetic problems in LLMs. Our interventions selectively alter the model's prediction at targeted digit positions, demonstrating the causal role of digit-position circuits in solving arithmetic tasks.