# An Empirical Investigation of Contextualized Number Prediction

arXiv: https://arxiv.org/abs/2011.07961 · HTML: https://mlsanity.com/paper/2011.07961/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Oct 2020.

## Built on by (most cited first, 21 of 21)

- [A Survey of Deep Learning for Mathematical Reasoning](https://mlsanity.com/paper/2212.10535/) (2022)
- [MultiHiertt: Numerical Reasoning over Multi Hierarchical Tabular and Textual Data](https://mlsanity.com/paper/2206.01347/) (2022)
- [Representing Numbers in NLP: a Survey and a Vision](https://mlsanity.com/paper/2103.13136/) (2021)
- [A Causal Framework to Quantify the Robustness of Mathematical Reasoning with Language Models](https://mlsanity.com/paper/2210.12023/) (2022)
- [Leveraging Language Foundation Models for Human Mobility Forecasting](https://mlsanity.com/paper/2209.05479/) (2022)
- [Can Language Beat Numerical Regression? Language-Based Multimodal Trajectory Prediction](https://mlsanity.com/paper/2403.18447/) (2024)
- [Mathematical Language Models: A Survey](https://mlsanity.com/paper/2312.07622/) (2023)
- [How Much Coffee Was Consumed During EMNLP 2019? Fermi Problems: A New Reasoning Challenge for AI](https://mlsanity.com/paper/2110.14207/) (2021)
- [NumGPT: Improving Numeracy Ability of Generative Pre-trained Models](https://mlsanity.com/paper/2109.03137/) (2021)
- [A Survey on Mathematical Reasoning and Optimization with Large Language Models](https://mlsanity.com/paper/2503.17726/) (2025)
- [Truth-Conditional Captioning of Time Series Data](https://mlsanity.com/paper/2110.01839/) (2021)
- [Detecting of a Patient's Condition From Clinical Narratives Using Natural Language Representation](https://mlsanity.com/paper/2104.03969/) (2021)
- [Masked Measurement Prediction: Learning to Jointly Predict Quantities and Units from Textual Context](https://mlsanity.com/paper/2112.08616/) (2021)
- [NAPG: Non-Autoregressive Program Generation for Hybrid Tabular-Textual Question Answering](https://mlsanity.com/paper/2211.03462/) (2022)
- [Materialized Knowledge Bases from Commonsense Transformers](https://mlsanity.com/paper/2112.14815/) (2021)
- [KitchenScale: Learning to predict ingredient quantities from recipe contexts](https://mlsanity.com/paper/2304.10739/) (2023)
- [Improving Downstream Task Performance by Treating Numbers as Entities](https://mlsanity.com/paper/2205.03559/) (2022)
- [Estimating Numbers without Regression](https://mlsanity.com/paper/2310.06204/) (2023)
- [Disambiguating Numeral Sequences to Decipher Ancient Accounting Corpora](https://mlsanity.com/paper/2502.00090/) (2025)
- [Wiki-Quantities and Wiki-Measurements: Datasets of Quantities and their Measurement Context from Wikipedia](https://mlsanity.com/paper/2503.14090/) (2025)
- [CONE: Embeddings for Complex Numerical Data Preserving Unit and Variable Semantics](https://mlsanity.com/paper/2603.04741/) (2026)

## Abstract (arXiv)

We conduct a large scale empirical investigation of contextualized number prediction in running text. Specifically, we consider two tasks: (1)masked number prediction-predicting a missing numerical value within a sentence, and (2)numerical anomaly detection-detecting an errorful numeric value within a sentence. We experiment with novel combinations of contextual encoders and output distributions over the real number line. Specifically, we introduce a suite of output distribution parameterizations that incorporate latent variables to add expressivity and better fit the natural distribution of numeric values in running text, and combine them with both recurrent and transformer-based encoder architectures. We evaluate these models on two numeric datasets in the financial and scientific domain. Our findings show that output distributions that incorporate discrete latent variables and allow for multiple modes outperform simple flow-based counterparts on all datasets, yielding more accurate numerical prediction and anomaly detection. We also show that our models effectively utilize textual con-text and benefit from general-purpose unsupervised pretraining.