mlsanityData to 14 Oct 2026

Language Models Encode Numbers Using Digit Representations in Base 10

arXiv 2410.11781 ↗First posted 15 Oct 2024cs.LGcited by 32

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) frequently make errors when handling even simple numerical problems, such as comparing two small numbers. A natural hypothesis is that these errors stem from how LLMs represent numbers, and specifically, whether their representations of numbers capture their numeric values. We tackle this question from the observation that LLM errors on numerical tasks are often distributed across the digits of the answer rather than normally around its numeric value. Through a series of probing experiments and causal interventions, we show that LLMs internally represent numbers with individual circular representations per-digit in base 10. This digit-wise representation, as opposed to a value representation, sheds light on the error patterns of models on tasks involving numerical reasoning and could serve as a basis for future studies on analyzing numerical mechanisms in LLMs.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Builds on 14 of its 31 references that are in the map. First posted on arXiv 15 Oct 2024.

Builds on

14 of 31 references.

References · 14

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  4. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  5. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
  6. 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
  7. 2024Not All Language Model Features Are One-Dimensionally Linearcited by 212
  8. 2019Do NLP Models Know Numbers? Probing Numeracy in Embeddingscited by 159
  9. 2021Investigating the Limitations of Transformers with Simple Arithmetic Taskscited by 139
  10. 2024Tokenization counts: the impact of tokenization on arithmetic in frontier LLMscited by 115
  11. 2023Successor Heads: Recurring, Interpretable Attention Heads In The Wildcited by 77
  12. 2024Pre-trained Large Language Models Use Fourier Features to Compute Additioncited by 39
2 more
  1. 2024When Can Transformers Count to n?cited by 24
  2. 2023Arithmetic with Language Models: from Memorization to Computationcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 17; 2026: 14.

32 papers.

Sort:

By area:

  1. 2025FoNE: Precise Single-Token Number Embeddings via Fourier Featurescited by 22
  2. 2025Probing for Arithmetic Errors in Language Modelscited by 18
  3. 2025The Lookahead Limitation: Why Multi-Operand Addition is Hard for LLMscited by 15
  4. 2025SOM Directions are Better than One: Multi-Directional Refusal Suppression in Language Modelscited by 9
  5. 2025Understanding In-context Learning of Addition via Activation Subspacescited by 8
  6. 2025Hypothesis-Driven Feature Manifold Analysis in LLMs via Supervised Multi-Dimensional Scalingcited by 8
  7. 2025Modular Arithmetic: Language Models Solve Math Digit by Digitcited by 6
  8. 2026Convergent Evolution: How Different Language Models Learn Similar Number Representationscited by 6
  9. 2025Constructing Interpretable Features from Compositional Neuron Groupscited by 5
  10. 2024Unraveling Arithmetic in Large Language Models: The Role of Algebraic Structurescited by 4
  11. 2025Number Representations in LLMs: A Computational Parallel to Human Perceptioncited by 4
  12. 2025Addition in Four Movements: Mapping Layer-wise Information Trajectories in LLMscited by 4
20 more
  1. 2025Pre-trained Language Models Learn Remarkably Accurate Representations of Numberscited by 4
  2. 2025Language Models Do Not Embed Numbers Continuouslycited by 4
  3. 2026Weber's Law in Transformer Magnitude Representations: Efficient Coding, Representational Geometry, and Psychophysical Laws in Language Modelscited by 4
  4. 2025UC-MOA: Utility-Conditioned Multi-Objective Alignment for Distributional Pareto-Optimalitycited by 2
  5. 2025InverseScope: Scalable Activation Inversion for Interpreting Large Language Modelscited by 2
  6. 2025StreetMath: Study of LLMs' Approximation Behaviorscited by 2
  7. 2025Language Models Learn Universal Representations of Numbers and Here's Why You Should Carecited by 2
  8. 2026Polar probe linearly decodes semantic structures from LLMscited by 2
  9. 2025Even Heads Fix Odd Errors: Mechanistic Discovery and Surgical Repair in Transformer Attentioncited by 1
  10. 2026AI-rithmeticcited by 1
  11. 2026Singular Vectors of Attention Heads Align with Featurescited by 1
  12. 2026Tool-Call Dependency Structure is Linearly Decodable in LLM Agent Residual Streamscited by 1
  13. 2026Numbers Already Carry Their Own Embeddingscited by 1
  14. 2026Greener Than Humans? Environmental Attitudes in Large Language Modelscited by 0
  15. 2026The Shape of Addition: Geometric Structures of Arithmetic in Large Language Modelscited by 0
  16. 2026Language Models Compare Quantities Using Number-specific and Unit-specific Heuristicscited by 0
  17. 2026When Tokenization is Secretly Output Supervisioncited by 0
  18. 2026The Geometry of Ignorance: LLMs Know When to Temper Bayesian Priorscited by 0
  19. 2026Causal and Interpretable Structures in LLM Compositional Taskscited by 0
  20. 2026Beyond Linear Concepts: Discovering and Aligning Non-Linear Concept Manifolds in Large Language Modelscited by 0