# How to Leverage Digit Embeddings to Represent Numbers?

arXiv: https://arxiv.org/abs/2407.00894 · HTML: https://mlsanity.com/paper/2407.00894/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 42 references that are in the map.
- First posted on arXiv 1 Jul 2024.

## Builds on (most cited first, 25 of 25)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Solving Quantitative Reasoning Problems with Language Models](https://mlsanity.com/paper/2206.14858/) (2022)
- [DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs](https://mlsanity.com/paper/1903.00161/) (2019)
- [MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms](https://mlsanity.com/paper/1905.13319/) (2019)
- [MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning](https://mlsanity.com/paper/2309.05653/) (2023)
- [Analysing Mathematical Reasoning Abilities of Neural Models](https://mlsanity.com/paper/1904.01557/) (2019)
- [Think before you speak: Training Language Models With Pause Tokens](https://mlsanity.com/paper/2310.02226/) (2023)
- [Do NLP Models Know Numbers? Probing Numeracy in Embeddings](https://mlsanity.com/paper/1909.07940/) (2019)
- [Injecting Numerical Reasoning Skills into Language Models](https://mlsanity.com/paper/2004.04487/) (2020)
- [Representing Numbers in NLP: a Survey and a Vision](https://mlsanity.com/paper/2103.13136/) (2021)
- [Do Language Embeddings Capture Scales?](https://mlsanity.com/paper/2010.05345/) (2020)
- [NumNet: Machine Reading Comprehension with Numerical Reasoning](https://mlsanity.com/paper/1910.06701/) (2019)
- [Numeracy for Language Models: Evaluating and Improving their Ability to Predict Numbers](https://mlsanity.com/paper/1805.08154/) (2018)
- [Seeking Patterns, Not just Memorizing Procedures: Contrastive Learning for Solving Math Word Problems](https://mlsanity.com/paper/2110.08464/) (2021)
- [Evaluating Transformer Language Models on Arithmetic Operations Using Number Decomposition](https://mlsanity.com/paper/2304.10977/) (2023)
- [Numeral Understanding in Financial Tweets for Fine-grained Crowd-based Forecasting](https://mlsanity.com/paper/1809.05356/) (2018)
- [Semantic-based Data Augmentation for Math Word Problems](https://mlsanity.com/paper/2201.02489/) (2022)
- [FERMAT: An Alternative to Accuracy for Numerical Reasoning](https://mlsanity.com/paper/2305.17491/) (2023)

## Built on by (most cited first, 5 of 5)

- [FoNE: Precise Single-Token Number Embeddings via Fourier Features](https://mlsanity.com/paper/2502.09741/) (2025)
- [Revealing the Numeracy Gap: An Empirical Investigation of Text Embedding Models](https://mlsanity.com/paper/2509.05691/) (2025)
- [Even GPT-5.2 Can't Count to Five: The Case for Zero-Error Horizons in Trustworthy LLMs](https://mlsanity.com/paper/2601.15714/) (2026)
- [Neural Isomorphic Fields: A Transformer-based Algebraic Numerical Embedding](https://mlsanity.com/paper/2601.12095/) (2026)
- [CONE: Embeddings for Complex Numerical Data Preserving Unit and Variable Semantics](https://mlsanity.com/paper/2603.04741/) (2026)

## Abstract (arXiv)

Within numerical reasoning, understanding numbers themselves is still a challenge for existing language models. Simple generalisations, such as solving 100+200 instead of 1+2, can substantially affect model performance (Sivakumar and Moosavi, 2023). Among various techniques, character-level embeddings of numbers have emerged as a promising approach to improve number representation. However, this method has limitations as it leaves the task of aggregating digit representations to the model, which lacks direct supervision for this process. In this paper, we explore the use of mathematical priors to compute aggregated digit embeddings and explicitly incorporate these aggregates into transformer models. This can be achieved either by adding a special token to the input embeddings or by introducing an additional loss function to enhance correct predictions. We evaluate the effectiveness of incorporating this explicit aggregation, analysing its strengths and shortcomings, and discuss future directions to better benefit from this approach. Our methods, while simple, are compatible with any pretrained model, easy to implement, and have been made publicly available.