# LLMs Know More About Numbers than They Can Say

arXiv: https://arxiv.org/abs/2602.07812 · HTML: https://mlsanity.com/paper/2602.07812/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 14 references that are in the map.
- First posted on arXiv 8 Feb 2026.

## Builds on (most cited first, 6 of 6)

- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [Galactica: A Large Language Model for Science](https://mlsanity.com/paper/2211.09085/) (2022)
- [Do NLP Models Know Numbers? Probing Numeracy in Embeddings](https://mlsanity.com/paper/1909.07940/) (2019)
- [Number Cookbook: Number Understanding of Language Models and How to Improve It](https://mlsanity.com/paper/2411.03766/) (2024)
- [Language Models Encode the Value of Numbers Linearly](https://mlsanity.com/paper/2401.03735/) (2024)

## Built on by (most cited first, 7 of 7)

- [RELISH: LLM REgression with a Latent Iterative State Head](https://mlsanity.com/paper/2604.01206/) (2026)
- [Reinforcement Learning over Predictive Distributions for LLM Regression](https://mlsanity.com/paper/2605.20740/) (2026)
- [LLM Sparsity Prior for Robust Feature Selection](https://mlsanity.com/paper/2605.23102/) (2026)
- [Language Models Compare Quantities Using Number-specific and Unit-specific Heuristics](https://mlsanity.com/paper/2606.03982/) (2026)
- [Wrong Prediction, Right Answer: Recovering Evidence from Collapsed LLM Sequence Scores](https://mlsanity.com/paper/2608.31068/) (2026)
- [When Models Don't Manipulate Manifolds: The Geometry of a Comparison Task](https://mlsanity.com/paper/2609.37680/) (2026)
- [DUALVIEW: Leveraging Broad and Output-Specific Evidence for LLM-based Multi-Output Text Regression](https://mlsanity.com/paper/2610.19206/) (2026)

## Abstract (arXiv)

Although state-of-the-art LLMs can solve math problems, we find that they make errors on numerical comparisons with mixed notation: "Which is larger, 5.7 × 102 or 580?" This raises a fundamental question: Do LLMs even know how big these numbers are? We probe the hidden states of several smaller open-source LLMs. A single linear projection of an appropriate hidden layer encodes the log-magnitudes of both kinds of numerals, allowing us to recover the numbers with relative error of about 2.3% (on restricted synthetic text) or 19.06% (on scientific papers). Furthermore, the hidden state after reading a pair of numerals encodes their ranking, with a linear classifier achieving over 90% accuracy. Yet surprisingly, when explicitly asked to rank the same pairs of numerals, these LLMs achieve only 50-70% accuracy, with worse performance for models whose probes are less effective. Finally, we show that incorporating the classifier probe's log-loss as an auxiliary objective during finetuning brings an additional 3.22% improvement in verbalized accuracy over base models, demonstrating that improving models' internal magnitude representations can enhance their numerical reasoning capabilities. Our code is available at https://github.com/VCY019/Numeracy-Probing.