# Revealing the Numeracy Gap: An Empirical Investigation of Text Embedding Models

arXiv: https://arxiv.org/abs/2509.05691 · HTML: https://mlsanity.com/paper/2509.05691/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 33 references that are in the map.
- First posted on arXiv 6 Sep 2025.

## Builds on (most cited first, 20 of 20)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models](https://mlsanity.com/paper/2506.05176/) (2025)
- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [MPNet: Masked and Permuted Pre-training for Language Understanding](https://mlsanity.com/paper/2004.09297/) (2020)
- [Towards General Text Embeddings with Multi-stage Contrastive Learning](https://mlsanity.com/paper/2308.03281/) (2023)
- [NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models](https://mlsanity.com/paper/2405.17428/) (2024)
- [SGPT: GPT Sentence Embeddings for Semantic Search](https://mlsanity.com/paper/2202.08904/) (2022)
- [Do NLP Models Know Numbers? Probing Numeracy in Embeddings](https://mlsanity.com/paper/1909.07940/) (2019)
- [MMTEB: Massive Multilingual Text Embedding Benchmark](https://mlsanity.com/paper/2502.13595/) (2025)
- [FinSQL: Model-Agnostic LLMs-based Text-to-SQL Framework for Financial Analysis](https://mlsanity.com/paper/2401.10506/) (2024)
- [Rare Tokens Degenerate All Tokens: Improving Neural Text Generation via Adaptive Gradient Gating for Rare Token Embeddings](https://mlsanity.com/paper/2109.03127/) (2021)
- [Learning Numeral Embeddings](https://mlsanity.com/paper/2001.00003/) (2019)
- [Beyond Surface Similarity: Detecting Subtle Semantic Shifts in Financial Narratives](https://mlsanity.com/paper/2403.14341/) (2024)
- [How to Leverage Digit Embeddings to Represent Numbers?](https://mlsanity.com/paper/2407.00894/) (2024)
- [Retrieval-augmented systems can be dangerous medical communicators](https://mlsanity.com/paper/2502.14898/) (2025)
- [FERMAT: An Alternative to Accuracy for Numerical Reasoning](https://mlsanity.com/paper/2305.17491/) (2023)
- [On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Survey](https://mlsanity.com/paper/2507.20783/) (2025)
- [Dense Retrievers Can Fail on Simple Queries: Revealing The Granularity Dilemma of Embeddings](https://mlsanity.com/paper/2506.08592/) (2025)

## Built on by (most cited first, 2 of 2)

- [Ricci-Filtration: Boosting Retrieval-Augmented Generation Reranker to Query-Answer Tasks by Discrete Ricci Flow](https://mlsanity.com/paper/2606.15482/) (2026)
- [MoMaTune: Model, Match, Then Tune Language Models for Numerical Records](https://mlsanity.com/paper/2610.15235/) (2026)

## Abstract (arXiv)

Text embedding models are widely used in natural language processing applications. However, their capability is often benchmarked on tasks that do not require understanding nuanced numerical information in text. As a result, it remains unclear whether current embedding models can precisely encode numerical content, such as numbers, into embeddings. This question is critical because embedding models are increasingly applied in domains where numbers matter, such as finance and healthcare. For example, Company X's market share grew by 2% should be interpreted very differently from Company X's market share grew by 20%, even though both indicate growth in market share. This study aims to examine whether text embedding models can capture such nuances. Using synthetic data in a financial context, we evaluate 13 widely used text embedding models and find that they generally struggle to capture numerical details accurately. Our further analyses provide deeper insights into embedding numeracy, informing future research to strengthen embedding model-based NLP systems with improved capacity for handling numerical content.