# The Shape of Addition: Geometric Structures of Arithmetic in Large Language Models

arXiv: https://arxiv.org/abs/2606.03645 · HTML: https://mlsanity.com/paper/2606.03645/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 34 of its 41 references that are in the map.
- First posted on arXiv 29 May 2026.

## Builds on (most cited first, 30 of 34)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023)
- [Not All Language Model Features Are One-Dimensionally Linear](https://mlsanity.com/paper/2405.14860/) (2024)
- [How well do Large Language Models perform in Arithmetic tasks?](https://mlsanity.com/paper/2304.02015/) (2023)
- [Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs](https://mlsanity.com/paper/2402.14903/) (2024)
- [Topology of deep neural networks](https://mlsanity.com/paper/2004.06093/) (2020)
- [Unsupervised Real-Time Hallucination Detection based on the Internal States of Large Language Models](https://mlsanity.com/paper/2403.06448/) (2024)
- [Language Models Use Trigonometry to Do Addition](https://mlsanity.com/paper/2502.00873/) (2025)
- [Eliciting Latent Knowledge from Quirky Language Models](https://mlsanity.com/paper/2312.01037/) (2023)
- [Interpreting and Improving Large Language Models in Arithmetic Calculation](https://mlsanity.com/paper/2409.01659/) (2024)
- [A Language Model's Guide Through Latent Space](https://mlsanity.com/paper/2402.14433/) (2024)
- [Language Models Encode Numbers Using Digit Representations in Base 10](https://mlsanity.com/paper/2410.11781/) (2024)
- [Towards Unifying Interpretability and Control: Evaluation via Intervention](https://mlsanity.com/paper/2411.04430/) (2024)
- [To Steer or Not to Steer? Mechanistic Error Reduction with Abstention for Language Models](https://mlsanity.com/paper/2510.13290/) (2025)
- [Language Models Encode the Value of Numbers Linearly](https://mlsanity.com/paper/2401.03735/) (2024)
- [LiSeCo: Linear Semantic Control for Language Generation](https://mlsanity.com/paper/2405.15454/) (2024)
- [Planning in a recurrent neural network that plays Sokoban](https://mlsanity.com/paper/2407.15421/) (2024)
- [Probing for Arithmetic Errors in Language Models](https://mlsanity.com/paper/2507.12379/) (2025)
- [The Lookahead Limitation: Why Multi-Operand Addition is Hard for LLMs](https://mlsanity.com/paper/2502.19981/) (2025)
- [Exposing Numeracy Gaps: A Benchmark to Evaluate Fundamental Numerical Abilities in Large Language Models](https://mlsanity.com/paper/2502.11075/) (2025)
- [The Geometries of Truth Are Orthogonal Across Tasks](https://mlsanity.com/paper/2506.08572/) (2025)
- [Understanding Addition and Subtraction in Transformers](https://mlsanity.com/paper/2402.02619/) (2024)
- [Hypothesis-Driven Feature Manifold Analysis in LLMs via Supervised Multi-Dimensional Scaling](https://mlsanity.com/paper/2510.01025/) (2025)
- [Dissecting Multiplication in Transformers: Insights into LLMs](https://mlsanity.com/paper/2407.15360/) (2024)

## Abstract (arXiv)

Large Language Models exhibit paradoxical fragility in fundamental arithmetic, implying a disconnect between internal computation and discrete output. By analyzing the residual stream geometry during multi-operand addition, we identify the Iso-Raw-Sum Trajectory (IRST), a geometric structure where representations are anchored by semantic digits and modulated by continuous carry fibers. We propose the Noisy Quantization Model to explain this geometry, framing arithmetic errors as Geometric Slippages caused by internal neural noise pushing a continuous, latent Carry Potential across quantization thresholds. This geometric framework further elucidates Probe Versatility, explaining how lightweight probes can disentangle coexisting latent signals (such as ground truth versus hallucination) from a single activation vector. Finally, we validate these insights through a geometric consistency check method that effectively detects and corrects these quantization failures during inference. Our code is available at https://github.com/RL-MIND/Shape-of-Addition.