# Language Models Do Not Embed Numbers Continuously

arXiv: https://arxiv.org/abs/2510.08009 · HTML: https://mlsanity.com/paper/2510.08009/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 21 references that are in the map.
- First posted on arXiv 9 Oct 2025.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [MedCalc-Bench: Evaluating Large Language Models for Medical Calculations](https://mlsanity.com/paper/2406.12036/) (2024)
- [Language Models Use Trigonometry to Do Addition](https://mlsanity.com/paper/2502.00873/) (2025)
- [Number Cookbook: Number Understanding of Language Models and How to Improve It](https://mlsanity.com/paper/2411.03766/) (2024)
- [Pre-trained Large Language Models Use Fourier Features to Compute Addition](https://mlsanity.com/paper/2406.03445/) (2024)
- [GeoGalactica: A Scientific Large Language Model in Geoscience](https://mlsanity.com/paper/2401.00434/) (2023)
- [Language Models Encode Numbers Using Digit Representations in Base 10](https://mlsanity.com/paper/2410.11781/) (2024)
- [NumeroLogic: Number Encoding for Enhanced LLMs' Numerical Reasoning](https://mlsanity.com/paper/2404.00459/) (2024)
- [What is a Number, That a Large Language Model May Know It?](https://mlsanity.com/paper/2502.01540/) (2025)
- [Arithmetic with Language Models: from Memorization to Computation](https://mlsanity.com/paper/2308.01154/) (2023)
- [Reverse That Number! Decoding Order Matters in Arithmetic Learning](https://mlsanity.com/paper/2403.05845/) (2024)
- [Leveraging Large Language Models for Medical Information Extraction and Query Generation](https://mlsanity.com/paper/2410.23851/) (2024)

## Built on by (most cited first, 4 of 4)

- [Can We Predict Before Executing Machine Learning Agents?](https://mlsanity.com/paper/2601.05930/) (2026)
- [PosterCopilot: Toward Layout Reasoning and Controllable Editing for Professional Graphic Design](https://mlsanity.com/paper/2512.04082/) (2025)
- [DriveCode: Domain Specific Numerical Encoding for LLM-Based Autonomous Driving](https://mlsanity.com/paper/2603.00919/) (2026)
- [Continuity and Ordinality Matter: Constraining Time Series Tokens for Effective Time Series Analysis with Large Language Models](https://mlsanity.com/paper/2605.28866/) (2026)

## Abstract (arXiv)

Recent research has extensively studied how large language models manipulate integers in specific arithmetic tasks, and on a more fundamental level, how they represent numeric values. These previous works have found that language model embeddings can be used to reconstruct the original values, however, they do not evaluate whether language models actually model continuous values as continuous. Using expected properties of the embedding space, including linear reconstruction and principal component analysis, we show that language models not only represent numeric spaces as non-continuous but also introduce significant noise. Using models from three major providers (OpenAI, Google Gemini and Voyage AI), we show that while reconstruction is possible with high fidelity (R2 ≥ 0.95), principal components only explain a minor share of variation within the embedding space. This indicates that many components within the embedding space are orthogonal to the simple numeric input space. Further, both linear reconstruction and explained variance suffer with increasing decimal precision, despite the ordinal nature of the input space being fundamentally unchanged. The findings of this work therefore have implications for the many areas where embedding models are used, in-particular where high numerical precision, large magnitudes or mixed-sign values are common.