# Estimating Numbers without Regression

arXiv: https://arxiv.org/abs/2310.06204 · HTML: https://mlsanity.com/paper/2310.06204/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 17 references that are in the map.
- First posted on arXiv 9 Oct 2023.

## Builds on (most cited first, 11 of 11)

- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks](https://mlsanity.com/paper/2211.12588/) (2022)
- [Knowledge Neurons in Pretrained Transformers](https://mlsanity.com/paper/2104.08696/) (2021)
- [Impact of Pretraining Term Frequencies on Few-Shot Reasoning](https://mlsanity.com/paper/2202.07206/) (2022)
- [Investigating the Limitations of Transformers with Simple Arithmetic Tasks](https://mlsanity.com/paper/2102.13019/) (2021)
- [Injecting Numerical Reasoning Skills into Language Models](https://mlsanity.com/paper/2004.04487/) (2020)
- [Representing Numbers in NLP: a Survey and a Vision](https://mlsanity.com/paper/2103.13136/) (2021)
- [How Much Coffee Was Consumed During EMNLP 2019? Fermi Problems: A New Reasoning Challenge for AI](https://mlsanity.com/paper/2110.14207/) (2021)
- [An Empirical Investigation of Contextualized Number Prediction](https://mlsanity.com/paper/2011.07961/) (2020)
- [Probing for Multilingual Numerical Understanding in Transformer-Based Language Models](https://mlsanity.com/paper/2010.06666/) (2020)

## Built on by (most cited first, 1 of 1)

- [A Triadic Suffix Tokenization Scheme for Numerical Reasoning](https://mlsanity.com/paper/2604.11582/) (2026)

## Abstract (arXiv)

Despite recent successes in language models, their ability to represent numbers is insufficient. Humans conceptualize numbers based on their magnitudes, effectively projecting them on a number line; whereas subword tokenization fails to explicitly capture magnitude by splitting numbers into arbitrary chunks. To alleviate this shortcoming, alternative approaches have been proposed that modify numbers at various stages of the language modeling pipeline. These methods change either the (1) notation in which numbers are written (eg scientific vs decimal), the (2) vocabulary used to represent numbers or the entire (3) architecture of the underlying language model, to directly regress to a desired number. Previous work suggests that architectural change helps achieve state-of-the-art on number estimation but we find an insightful ablation: changing the model's vocabulary instead (eg introduce a new token for numbers in range 10-100) is a far better trade-off. In the context of masked number prediction, a carefully designed tokenization scheme is both the simplest to implement and sufficient, ie with similar performance to the state-of-the-art approach that requires making significant architectural changes. Finally, we report similar trends on the downstream task of numerical fact estimation (for Fermi Problems) and discuss reasons behind our findings.