# Language Models Encode Numbers Using Digit Representations in Base 10

arXiv: https://arxiv.org/abs/2410.11781 · HTML: https://mlsanity.com/paper/2410.11781/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 31 references that are in the map.
- First posted on arXiv 15 Oct 2024.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021)
- [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023)
- [Not All Language Model Features Are One-Dimensionally Linear](https://mlsanity.com/paper/2405.14860/) (2024)
- [Do NLP Models Know Numbers? Probing Numeracy in Embeddings](https://mlsanity.com/paper/1909.07940/) (2019)
- [Investigating the Limitations of Transformers with Simple Arithmetic Tasks](https://mlsanity.com/paper/2102.13019/) (2021)
- [Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs](https://mlsanity.com/paper/2402.14903/) (2024)
- [Successor Heads: Recurring, Interpretable Attention Heads In The Wild](https://mlsanity.com/paper/2312.09230/) (2023)
- [Pre-trained Large Language Models Use Fourier Features to Compute Addition](https://mlsanity.com/paper/2406.03445/) (2024)
- [When Can Transformers Count to n?](https://mlsanity.com/paper/2407.15160/) (2024)
- [Arithmetic with Language Models: from Memorization to Computation](https://mlsanity.com/paper/2308.01154/) (2023)

## Built on by (most cited first, 30 of 32)

- [FoNE: Precise Single-Token Number Embeddings via Fourier Features](https://mlsanity.com/paper/2502.09741/) (2025)
- [Probing for Arithmetic Errors in Language Models](https://mlsanity.com/paper/2507.12379/) (2025)
- [The Lookahead Limitation: Why Multi-Operand Addition is Hard for LLMs](https://mlsanity.com/paper/2502.19981/) (2025)
- [SOM Directions are Better than One: Multi-Directional Refusal Suppression in Language Models](https://mlsanity.com/paper/2511.08379/) (2025)
- [Understanding In-context Learning of Addition via Activation Subspaces](https://mlsanity.com/paper/2505.05145/) (2025)
- [Hypothesis-Driven Feature Manifold Analysis in LLMs via Supervised Multi-Dimensional Scaling](https://mlsanity.com/paper/2510.01025/) (2025)
- [Modular Arithmetic: Language Models Solve Math Digit by Digit](https://mlsanity.com/paper/2508.02513/) (2025)
- [Convergent Evolution: How Different Language Models Learn Similar Number Representations](https://mlsanity.com/paper/2604.20817/) (2026)
- [Constructing Interpretable Features from Compositional Neuron Groups](https://mlsanity.com/paper/2506.10920/) (2025)
- [Unraveling Arithmetic in Large Language Models: The Role of Algebraic Structures](https://mlsanity.com/paper/2411.16260/) (2024)
- [Number Representations in LLMs: A Computational Parallel to Human Perception](https://mlsanity.com/paper/2502.16147/) (2025)
- [Addition in Four Movements: Mapping Layer-wise Information Trajectories in LLMs](https://mlsanity.com/paper/2506.07824/) (2025)
- [Pre-trained Language Models Learn Remarkably Accurate Representations of Numbers](https://mlsanity.com/paper/2506.08966/) (2025)
- [Language Models Do Not Embed Numbers Continuously](https://mlsanity.com/paper/2510.08009/) (2025)
- [Weber's Law in Transformer Magnitude Representations: Efficient Coding, Representational Geometry, and Psychophysical Laws in Language Models](https://mlsanity.com/paper/2603.20642/) (2026)
- [UC-MOA: Utility-Conditioned Multi-Objective Alignment for Distributional Pareto-Optimality](https://mlsanity.com/paper/2503.10669/) (2025)
- [InverseScope: Scalable Activation Inversion for Interpreting Large Language Models](https://mlsanity.com/paper/2506.07406/) (2025)
- [StreetMath: Study of LLMs' Approximation Behaviors](https://mlsanity.com/paper/2510.25776/) (2025)
- [Language Models Learn Universal Representations of Numbers and Here's Why You Should Care](https://mlsanity.com/paper/2510.26285/) (2025)
- [Polar probe linearly decodes semantic structures from LLMs](https://mlsanity.com/paper/2605.14125/) (2026)
- [Even Heads Fix Odd Errors: Mechanistic Discovery and Surgical Repair in Transformer Attention](https://mlsanity.com/paper/2508.19414/) (2025)
- [AI-rithmetic](https://mlsanity.com/paper/2602.10416/) (2026)
- [Singular Vectors of Attention Heads Align with Features](https://mlsanity.com/paper/2602.13524/) (2026)
- [Tool-Call Dependency Structure is Linearly Decodable in LLM Agent Residual Streams](https://mlsanity.com/paper/2605.25310/) (2026)
- [Numbers Already Carry Their Own Embeddings](https://mlsanity.com/paper/2606.14108/) (2026)
- [Greener Than Humans? Environmental Attitudes in Large Language Models](https://mlsanity.com/paper/2606.02741/) (2026)
- [The Shape of Addition: Geometric Structures of Arithmetic in Large Language Models](https://mlsanity.com/paper/2606.03645/) (2026)
- [Language Models Compare Quantities Using Number-specific and Unit-specific Heuristics](https://mlsanity.com/paper/2606.03982/) (2026)
- [When Tokenization is Secretly Output Supervision](https://mlsanity.com/paper/2609.01386/) (2026)
- [The Geometry of Ignorance: LLMs Know When to Temper Bayesian Priors](https://mlsanity.com/paper/2609.02959/) (2026)

## Abstract (arXiv)

Large language models (LLMs) frequently make errors when handling even simple numerical problems, such as comparing two small numbers. A natural hypothesis is that these errors stem from how LLMs represent numbers, and specifically, whether their representations of numbers capture their numeric values. We tackle this question from the observation that LLM errors on numerical tasks are often distributed across the digits of the answer rather than normally around its numeric value. Through a series of probing experiments and causal interventions, we show that LLMs internally represent numbers with individual circular representations per-digit in base 10. This digit-wise representation, as opposed to a value representation, sheds light on the error patterns of models on tasks involving numerical reasoning and could serve as a basis for future studies on analyzing numerical mechanisms in LLMs.