# Language Models Learn Universal Representations of Numbers and Here's Why You Should Care

arXiv: https://arxiv.org/abs/2510.26285 · HTML: https://mlsanity.com/paper/2510.26285/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 26 references that are in the map.
- First posted on arXiv 30 Oct 2025.

## Builds on (most cited first, 18 of 18)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023)
- [Eliciting Latent Predictions from Transformers with the Tuned Lens](https://mlsanity.com/paper/2303.08112/) (2023)
- [Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspective](https://mlsanity.com/paper/2305.15408/) (2023)
- [Not All Language Model Features Are One-Dimensionally Linear](https://mlsanity.com/paper/2405.14860/) (2024)
- [Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristics](https://mlsanity.com/paper/2410.21272/) (2024)
- [Language Models Use Trigonometry to Do Addition](https://mlsanity.com/paper/2502.00873/) (2025)
- [LatestEval: Addressing Data Contamination in Language Model Evaluation through Dynamic and Time-Sensitive Test Construction](https://mlsanity.com/paper/2312.12343/) (2023)
- [Interpreting and Improving Large Language Models in Arithmetic Calculation](https://mlsanity.com/paper/2409.01659/) (2024)
- [Benchmarking Hallucination in Large Language Models based on Unanswerable Math Word Problem](https://mlsanity.com/paper/2403.03558/) (2024)
- [Pre-trained Large Language Models Use Fourier Features to Compute Addition](https://mlsanity.com/paper/2406.03445/) (2024)
- [Language Models Encode Numbers Using Digit Representations in Base 10](https://mlsanity.com/paper/2410.11781/) (2024)
- [How to Dissect a Muppet: The Structure of Transformer Embedding Spaces](https://mlsanity.com/paper/2206.03529/) (2022)
- [Language Models Encode the Value of Numbers Linearly](https://mlsanity.com/paper/2401.03735/) (2024)
- [Proving Olympiad Inequalities by Synergizing LLMs and Symbolic Reasoning](https://mlsanity.com/paper/2502.13834/) (2025)
- [Pre-trained Language Models Learn Remarkably Accurate Representations of Numbers](https://mlsanity.com/paper/2506.08966/) (2025)

## Built on by (most cited first, 2 of 2)

- [Weber's Law in Transformer Magnitude Representations: Efficient Coding, Representational Geometry, and Psychophysical Laws in Language Models](https://mlsanity.com/paper/2603.20642/) (2026)
- [Dynamic Routing as a New Dimension for Test-time Versatility of LLMs](https://mlsanity.com/paper/2610.04751/) (2026)

## Abstract (arXiv)

Prior work has shown that large language models (LLMs) often converge to accurate input embedding for numbers, based on sinusoidal representations. In this work, we quantify that these representations are in fact strikingly systematic, to the point of being almost perfectly universal: different LLM families develop equivalent sinusoidal structures, and number representations are broadly interchangeable in a large swathe of experimental setups. We show that properly factoring in this characteristic is crucial when it comes to assessing how accurately LLMs encode numeric and other ordinal information, and that mechanistically enhancing this sinusoidality can also lead to reductions of LLMs' arithmetic errors.