mlsanityData to 7 Oct 2026

Representing Numbers in NLP: a Survey and a Vision

arXiv 2103.13136 ↗First posted 24 Mar 2021cs.CL cs.AI cs.LGcited by 83

Not on the map. Why?

Abstract (arXiv)

NLP systems rarely give special consideration to numbers found in text. This starkly contrasts with the consensus in neuroscience that, in the brain, numbers are represented differently from words. We arrange recent NLP work on numeracy into a comprehensive taxonomy of tasks and methods. We break down the subjective notion of numeracy into 7 subtasks, arranged along two dimensions: granularity (exact vs approximate) and units (abstract vs grounded). We analyze the myriad representational choices made by 18 previously published number encoders and decoders. We synthesize best practices for representing numbers in text and articulate a vision for holistic numeracy in NLP, comprised of design trade-offs and a unified evaluation.

Key facts. Cited by 83 papers in the map (data through 7 Oct 2026). Builds on 41 of its 70 references that are in the map. First posted on arXiv 24 Mar 2021.

Builds on

41 of 70 references.

References · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  9. 2018Deep contextualized word representationscited by 2,285
  10. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  11. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  12. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
29 more
  1. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  2. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  3. 2016Solving General Arithmetic Word Problemscited by 353
  4. 2019Analysing Mathematical Reasoning Abilities of Neural Modelscited by 331
  5. 2016Using the Output Embedding to Improve Language Modelscited by 266
  6. 2019Do NLP Models Know Numbers? Probing Numeracy in Embeddingscited by 157
  7. 2021Investigating the Limitations of Transformers with Simple Arithmetic Taskscited by 136
  8. 2020Injecting Numerical Reasoning Skills into Language Modelscited by 133
  9. 2020Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-trained Language Modelscited by 110
  10. 2020On the Computational Power of Transformers and its Implications in Sequence Modelingcited by 85
  11. 2018Neural Arithmetic Logic Unitscited by 69
  12. 2020Do Language Embeddings Capture Scales?cited by 63
  13. 2019NumNet: Machine Reading Comprehension with Numerical Reasoningcited by 59
  14. 2019Giving BERT a Calculator: Finding Operations and Arguments with Reading Comprehensioncited by 51
  15. 2018QuaRel: A Dataset and Models for Answering Questions about Qualitative Relationshipscited by 46
  16. 2019LSTM Networks Can Perform Dynamic Countingcited by 45
  17. 2018Numeracy for Language Models: Evaluating and Improving their Ability to Predict Numberscited by 42
  18. 2019EQUATE: A Benchmark Evaluation Framework for Quantitative Reasoning in Natural Language Inferencecited by 39
  19. 2020Temporal Common Sense Acquisition with Minimal Supervisioncited by 38
  20. 2019From 'F' to 'A' on the N.Y. Regents Science Exams: An Overview of the Aristo Projectcited by 33
  21. 2017Verb Physics: Relative Physical Knowledge of Actions and Objectscited by 26
  22. 2019How Large Are Lions? Inducing Distributions over Quantitative Attributescited by 26
  23. 2020An Empirical Investigation of Contextualized Number Predictioncited by 21
  24. 2019Learning Numeral Embeddingscited by 15
  25. 2019Where's My Head? Definition, Dataset and Models for Numeric Fused-Heads Identification and Resolutioncited by 11
  26. 2020Towards Question Format Independent Numerical Reasoning: A Set of Prerequisite Taskscited by 10
  27. 2020Probing for Multilingual Numerical Understanding in Transformer-Based Language Modelscited by 10
  28. 2018Numeral Understanding in Financial Tweets for Fine-grained Crowd-based Forecastingcited by 6
  29. 2020Exploring Early Prediction of Buyer-Seller Negotiation Outcomescited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 20; 2023: 16; 2024: 16; 2025: 13; 2026: 9.

83 papers.

Sort:

By area:

  1. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  2. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  3. 2023How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelcited by 199
  4. 2022mGPT: Few-Shot Learners Go Multilingualcited by 141
  5. 2022A Survey of Deep Learning for Mathematical Reasoningcited by 124
  6. 2021HiTab: A Hierarchical Table Dataset for Question Answering and Natural Language Generationcited by 117
  7. 2022Teaching Algorithmic Reasoning via In-context Learningcited by 111
  8. 2024Why Tabular Foundation Models Should Be a Research Prioritycited by 103
  9. 2022A Causal Framework to Quantify the Robustness of Mathematical Reasoning with Language Modelscited by 68
  10. 2023MLCopilot: Unleashing the Power of Large Language Models in Solving Machine Learning Taskscited by 66
  11. 2024Getting the most out of your tokenizer for pre-training and domain adaptationcited by 65
  12. 2025SuperBPE: Space Travel for Language Modelscited by 58
38 more
  1. 2022FiNER: Financial Numeric Entity Recognition for XBRL Taggingcited by 47
  2. 2022PACIFIC: Towards Proactive Conversational Question Answering over Tabular and Textual Data in Financecited by 45
  3. 2022OrdinalCLIP: Learning Rank Prompts for Language-Guided Ordinal Regressioncited by 44
  4. 2024Interpreting and Improving Large Language Models in Arithmetic Calculationcited by 43
  5. 2022Table Pre-training: A Survey on Model Architectures, Pre-training Objectives, and Downstream Taskscited by 42
  6. 2022T2G-Former: Organizing Tabular Features into Relation Graphs Promotes Heterogeneous Feature Interactioncited by 41
  7. 2023SNIP: Bridging Mathematical Symbolic and Numeric Realms with Unified Pre-trainingcited by 41
  8. 2021Turning Tables: Generating Examples from Semi-structured Tables for Endowing Language Models with Reasoning Skillscited by 39
  9. 2024Pre-trained Large Language Models Use Fourier Features to Compute Additioncited by 38
  10. 2023Reading and Reasoning over Chart Images for Evidence-based Automated Fact-Checkingcited by 34
  11. 2021MWP-BERT: Numeracy-Augmented Pre-training for Math Word Problem Solvingcited by 32
  12. 2023LLM-Prop: Predicting Physical And Electronic Properties Of Crystalline Solids From Their Text Descriptionscited by 29
  13. 2024Large Language Models in Drug Discovery and Development: From Disease Mechanisms to Clinical Trialscited by 24
  14. 2025FoNE: Precise Single-Token Number Embeddings via Fourier Featurescited by 21
  15. 2023Can neural networks do arithmetic? A survey on the elementary numerical skills of state-of-the-art deep learning modelscited by 18
  16. 2025Broken Tokens? Your Language Model can Secretly Handle Non-Canonical Tokenizationscited by 16
  17. 2021NumGPT: Improving Numeracy Ability of Generative Pre-trained Modelscited by 15
  18. 2023Evaluating Transformer Language Models on Arithmetic Operations Using Number Decompositioncited by 15
  19. 2025TabSTAR: A Tabular Foundation Model for Tabular Data with Text Fieldscited by 15
  20. 2025Orion-MSP: Multi-Scale Sparse Attention for Tabular In-Context Learningcited by 15
  21. 2022A Survey on Medical Document Summarizationcited by 14
  22. 2024Interpretable Vision-Language Survival Analysis with Ordinal Inductive Bias for Computational Pathologycited by 14
  23. 2021Investigating Numeracy Learning Ability of a Text-to-Text Transfer Modelcited by 13
  24. 2024Regress, Don't Guess -- A Regression-like Loss on Number Tokens for Language Modelscited by 12
  25. 2025Tokenization is Sensitive to Language Variationcited by 12
  26. 2021Truth-Conditional Captioning of Time Series Datacited by 11
  27. 2024Language Models are Symbolic Learners in Arithmeticcited by 11
  28. 2022Innovations in Neural Data-to-text Generation: A Surveycited by 10
  29. 2022Induced Natural Language Rationales and Interleaved Markup Tokens Enable Extrapolation in Large Language Modelscited by 10
  30. 2023Human Behavioral Benchmarking: Numeric Magnitude Comparison Effects in Large Language Modelscited by 10
  31. 2024Explicitly Encoding Structural Symmetry is Key to Length Generalization in Arithmetic Taskscited by 10
  32. 2021Numerical reasoning in machine reading comprehension tasks: are we there yet?cited by 9
  33. 2022Developing a general-purpose clinical language inference model from a large corpus of clinical notescited by 8
  34. 2022LUNA: Language Understanding with Number Augmentations on Transformers via Number Plugins and Pre-trainingcited by 8
  35. 2023Exploring Equation as a Better Intermediate Meaning Representation for Numerical Reasoningcited by 7
  36. 2023Learn Your Tokens: Word-Pooled Tokenization for Language Modelingcited by 7
  37. 2025In-context Learning of Evolving Data Streams with Tabular Foundational Modelscited by 7
  38. 2022Towards Robust Numerical Question Answering: Diagnosing Numerical Capabilities of NLP Systemscited by 6

50 most cited of 83 All 83 (JSON)