# Language Models Encode the Value of Numbers Linearly

arXiv: https://arxiv.org/abs/2401.03735 · HTML: https://mlsanity.com/paper/2401.03735/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in Steering and interpreting LLM reasoning, one of 13 directions in Mechanistic interpretability.
- Builds on 15 of its 32 references that are in the map.
- First posted on arXiv 8 Jan 2024.

## Placed in

- Mechanistic interpretability / [Steering and interpreting LLM reasoning](https://mlsanity.com/mechinterp/steering-and-interpreting-llm-reasoning/)

## Builds on (most cited first, 15 of 15)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis](https://mlsanity.com/paper/2203.13474/) (2022)
- [Faith and Fate: Limits of Transformers on Compositionality](https://mlsanity.com/paper/2305.18654/) (2023)
- [What do you learn from context? Probing for sentence structure in contextualized word representations](https://mlsanity.com/paper/1905.06316/) (2019)
- [Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task](https://mlsanity.com/paper/2210.13382/) (2022)
- [Emergent Linear Representations in World Models of Self-Supervised Sequence Models](https://mlsanity.com/paper/2309.00941/) (2023)
- [Implicit Representations of Meaning in Neural Language Models](https://mlsanity.com/paper/2106.00737/) (2021)
- [Do NLP Models Know Numbers? Probing Numeracy in Embeddings](https://mlsanity.com/paper/1909.07940/) (2019)
- [The Hydra Effect: Emergent Self-repair in Language Model Computations](https://mlsanity.com/paper/2307.15771/) (2023)
- [Verify-and-Edit: A Knowledge-Enhanced Chain-of-Thought Framework](https://mlsanity.com/paper/2305.03268/) (2023)

## Built on by (most cited first, 20 of 20)

- [Probing for Arithmetic Errors in Language Models](https://mlsanity.com/paper/2507.12379/) (2025)
- [What is a Number, That a Large Language Model May Know It?](https://mlsanity.com/paper/2502.01540/) (2025)
- [The Lookahead Limitation: Why Multi-Operand Addition is Hard for LLMs](https://mlsanity.com/paper/2502.19981/) (2025)
- [LLMs Faithfully and Iteratively Compute Answers During CoT: A Systematic Analysis With Multi-step Arithmetics](https://mlsanity.com/paper/2412.01113/) (2024)
- [LLMs Know More About Numbers than They Can Say](https://mlsanity.com/paper/2602.07812/) (2026)
- [Modular Arithmetic: Language Models Solve Math Digit by Digit](https://mlsanity.com/paper/2508.02513/) (2025)
- [Number Representations in LLMs: A Computational Parallel to Human Perception](https://mlsanity.com/paper/2502.16147/) (2025)
- [Addition in Four Movements: Mapping Layer-wise Information Trajectories in LLMs](https://mlsanity.com/paper/2506.07824/) (2025)
- [Pre-trained Language Models Learn Remarkably Accurate Representations of Numbers](https://mlsanity.com/paper/2506.08966/) (2025)
- [Language Models Learn Universal Representations of Numbers and Here's Why You Should Care](https://mlsanity.com/paper/2510.26285/) (2025)
- [Categorical Perception in Large Language Model Hidden States: Structural Warping at Digit-Count Boundaries](https://mlsanity.com/paper/2603.28258/) (2026)
- [On Representational Dissociation of Language and Arithmetic in Large Language Models](https://mlsanity.com/paper/2502.11932/) (2025)
- [Univariate to Multivariate: LLMs as Zero-Shot Predictors for Time-Series Forecasting](https://mlsanity.com/paper/2506.02389/) (2025)
- [DriveCode: Domain Specific Numerical Encoding for LLM-Based Autonomous Driving](https://mlsanity.com/paper/2603.00919/) (2026)
- [The Shape of Addition: Geometric Structures of Arithmetic in Large Language Models](https://mlsanity.com/paper/2606.03645/) (2026)
- [Language Models Compare Quantities Using Number-specific and Unit-specific Heuristics](https://mlsanity.com/paper/2606.03982/) (2026)
- [Prime Fourier Embeddings: A Principled Basis for Modular Arithmetic](https://mlsanity.com/paper/2606.23044/) (2026)
- [The Geometry of Ignorance: LLMs Know When to Temper Bayesian Priors](https://mlsanity.com/paper/2609.02959/) (2026)
- [The Advantages of Fresh Sketching for Ridge Regression](https://mlsanity.com/paper/2609.38565/) (2026)
- [RepProbe: Can Video Language Models Count Repeated Actions? A Diagnostic Benchmark](https://mlsanity.com/paper/2610.18844/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have exhibited impressive competence in various tasks, but their internal mechanisms on mathematical problems are still under-explored. In this paper, we study a fundamental question: how language models encode the value of numbers, a basic element in math. To study the question, we construct a synthetic dataset comprising addition problems and utilize linear probes to read out input numbers from the hidden states. Experimental results support the existence of encoded number values in LLMs on different layers, and these values can be extracted via linear probes. Further experiments show that LLMs store their calculation results in a similar manner, and we can intervene the output via simple vector additions, proving the causal connection between encoded numbers and language model outputs. Our research provides evidence that LLMs encode the value of numbers linearly, offering insights for better exploring, designing, and utilizing numeric information in LLMs.