# Efficient numeracy in language models through single-token number embeddings

arXiv: https://arxiv.org/abs/2510.06824 · HTML: https://mlsanity.com/paper/2510.06824/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 57 of its 80 references that are in the map.
- First posted on arXiv 8 Oct 2025.

## Builds on (most cited first, 30 of 57)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters](https://mlsanity.com/paper/2408.03314/) (2024)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [gpt-oss-120b & gpt-oss-20b Model Card](https://mlsanity.com/paper/2508.10925/) (2025)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale](https://mlsanity.com/paper/2406.17557/) (2024)
- [Training Large Language Models to Reason in a Continuous Latent Space](https://mlsanity.com/paper/2412.06769/) (2024)
- [PAL: Program-aided Language Models](https://mlsanity.com/paper/2211.10435/) (2022)
- [Kimi K2: Open Agentic Intelligence](https://mlsanity.com/paper/2507.20534/) (2025)
- [GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models](https://mlsanity.com/paper/2410.05229/) (2024)
- [CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning](https://mlsanity.com/paper/2207.01780/) (2022)
- [Demystifying Long Chain-of-Thought Reasoning in LLMs](https://mlsanity.com/paper/2502.03373/) (2025)
- [A Long Way to Go: Investigating Length Correlations in RLHF](https://mlsanity.com/paper/2310.03716/) (2023)
- [Large Language Models for Mathematical Reasoning: Progresses and Challenges](https://mlsanity.com/paper/2402.00157/) (2024)
- [ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving](https://mlsanity.com/paper/2309.17452/) (2023)
- [TabLLM: Few-shot Classification of Tabular Data with Large Language Models](https://mlsanity.com/paper/2210.10723/) (2022)
- [Tool Learning with Large Language Models: A Survey](https://mlsanity.com/paper/2405.17935/) (2024)
- [TALM: Tool Augmented Language Models](https://mlsanity.com/paper/2205.12255/) (2022)
- [Chain-of-Table: Evolving Tables in the Reasoning Chain for Table Understanding](https://mlsanity.com/paper/2401.04398/) (2024)
- [Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Survey](https://mlsanity.com/paper/2402.17944/) (2024)

## Built on by (most cited first, 7 of 7)

- [The Illusion of Stochasticity in LLMs](https://mlsanity.com/paper/2604.06543/) (2026)
- [The Effect of Scripts and Formats on LLM Numeracy](https://mlsanity.com/paper/2601.15251/) (2026)
- [What Really Controls Temporal Reasoning in Large Language Models: Tokenisation or Representation of Time?](https://mlsanity.com/paper/2603.19017/) (2026)
- [A Triadic Suffix Tokenization Scheme for Numerical Reasoning](https://mlsanity.com/paper/2604.11582/) (2026)
- [Large Language Models as Amortized Pareto-Front Generators for Constrained Bi-Objective Convex Optimization](https://mlsanity.com/paper/2605.12106/) (2026)
- [The Importance of Being Statistically Earnest: A Critical Re-evaluation of GSM-Symbolic](https://mlsanity.com/paper/2605.28700/) (2026)
- [From Fragments to Global Maps: Learning Vectorized Map Aggregation with Large Language Models](https://mlsanity.com/paper/2610.02513/) (2026)

## Abstract (arXiv)

To drive progress in science and engineering, large language models (LLMs) must be able to process large amounts of numerical data and solve long calculations efficiently. This is currently only possible through the use of external tools or extensive reasoning chains, either weakening the numerical representations of LLMs or limiting the length of problems they can solve. We show that frontier LLMs require excessive amounts of reasoning tokens to solve even basic calculations, which is exacerbated by their tokenization strategies that split single numbers into multiple tokens. This motivates the need for efficient and effective single-token number encodings. We introduce a set of desiderata for such encodings and show that existing approaches fail to fulfill them. To address these shortcomings, we propose BitTokens, a novel encoding strategy that represents any number as a single token using its IEEE 754 binary floating-point representation. Through extensive experiments we show that our BitTokens allow even small language models to learn algorithms that solve basic arithmetic operations nearly perfectly. This newly gained efficiency could expand the length and complexity of problems language models can solve.