# Learning Numeral Embeddings

arXiv: https://arxiv.org/abs/2001.00003 · HTML: https://mlsanity.com/paper/2001.00003/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Dec 2019.

## Built on by (most cited first, 15 of 15)

- [Why Tabular Foundation Models Should Be a Research Priority](https://mlsanity.com/paper/2405.01147/) (2024)
- [Language Model Behavior: A Comprehensive Survey](https://mlsanity.com/paper/2303.11504/) (2023)
- [Representing Numbers in NLP: a Survey and a Vision](https://mlsanity.com/paper/2103.13136/) (2021)
- [Can neural networks do arithmetic? A survey on the elementary numerical skills of state-of-the-art deep learning models](https://mlsanity.com/paper/2303.07735/) (2023)
- [Graph Neural Networks and Representation Embedding for Table Extraction in PDF Documents](https://mlsanity.com/paper/2208.11203/) (2022)
- [Learning Non-linguistic Skills without Sacrificing Linguistic Proficiency](https://mlsanity.com/paper/2305.08246/) (2023)
- [Overcoming Barriers to Skill Injection in Language Modeling: Case Study in Arithmetic](https://mlsanity.com/paper/2211.02098/) (2022)
- [Revealing the Numeracy Gap: An Empirical Investigation of Text Embedding Models](https://mlsanity.com/paper/2509.05691/) (2025)
- [Laying Anchors: Semantically Priming Numerals in Language Modeling](https://mlsanity.com/paper/2404.01536/) (2024)
- [Numbers Matter! Bringing Quantity-awareness to Retrieval Systems](https://mlsanity.com/paper/2407.10283/) (2024)
- [Gem: Gaussian Mixture Model Embeddings for Numerical Feature Distributions](https://mlsanity.com/paper/2410.07485/) (2024)
- [Exploring Internal Numeracy in Language Models: A Case Study on ALBERT](https://mlsanity.com/paper/2404.16574/) (2024)
- [Neural Isomorphic Fields: A Transformer-based Algebraic Numerical Embedding](https://mlsanity.com/paper/2601.12095/) (2026)
- [CONE: Embeddings for Complex Numerical Data Preserving Unit and Variable Semantics](https://mlsanity.com/paper/2603.04741/) (2026)
- [PyraMathBench: Evaluating and Improving Mathematical Capability in Large Language Models](https://mlsanity.com/paper/2606.03858/) (2026)

## Abstract (arXiv)

Word embedding is an essential building block for deep learning methods for natural language processing. Although word embedding has been extensively studied over the years, the problem of how to effectively embed numerals, a special subset of words, is still underexplored. Existing word embedding methods do not learn numeral embeddings well because there are an infinite number of numerals and their individual appearances in training corpora are highly scarce. In this paper, we propose two novel numeral embedding methods that can handle the out-of-vocabulary (OOV) problem for numerals. We first induce a finite set of prototype numerals using either a self-organizing map or a Gaussian mixture model. We then represent the embedding of a numeral as a weighted average of the prototype number embeddings. Numeral embeddings represented in this manner can be plugged into existing word embedding learning approaches such as skip-gram for training. We evaluated our methods and showed its effectiveness on four intrinsic and extrinsic tasks: word similarity, embedding numeracy, numeral prediction, and sequence labeling.