mlsanityData to 7 Oct 2026

Do NLP Models Know Numbers? Probing Numeracy in Embeddings

arXiv 1909.07940 ↗First posted 17 Sep 2019cs.CL cs.LGcited by 157

Not on the map. Why?

Cited by 20 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The ability to understand and work with numbers (numeracy) is critical for many complex reasoning tasks. Currently, most NLP models treat numbers in text in the same way as other tokens---they embed them as distributed vectors. Is this enough to capture numeracy? We begin by investigating the numerical reasoning capabilities of a state-of-the-art question answering model on the DROP dataset. We find this model excels on questions that require numerical reasoning, i.e., it already captures numeracy. To understand how this capability emerges, we probe token embedding methods (e.g., BERT, GloVe) on synthetic list maximum, number decoding, and addition tasks. A surprising degree of numeracy is naturally present in standard embeddings. For example, GloVe and word2vec accurately encode magnitude for numbers up to 1,000. Furthermore, character-level embeddings are even more precise---ELMo captures numeracy the best for all pre-trained methods---but BERT, which uses sub-word units, is less exact.

Key facts. Cited by 157 papers in the map (data through 7 Oct 2026). First posted on arXiv 17 Sep 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 33; 2022: 31; 2023: 26; 2024: 23; 2025: 19; 2026: 25.

157 papers.

Sort:

By area:

  1. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  2. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  3. 2022Language Models are Realistic Tabular Data Generatorscited by 290
  4. 2023How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelcited by 199
  5. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attackscited by 189
  6. 2022Probing Pre-Trained Language Models for Cross-Cultural Differences in Valuescited by 147
  7. 2022LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Taskscited by 146
  8. 2022Impact of Pretraining Term Frequencies on Few-Shot Reasoningcited by 142
  9. 2021Investigating the Limitations of Transformers with Simple Arithmetic Taskscited by 136
  10. 2022Towards Faithful Model Explanation in NLP: A Surveycited by 129
  11. 2022A Survey of Deep Learning for Mathematical Reasoningcited by 124
  12. 2023Teaching Arithmetic to Small Transformerscited by 121
38 more
  1. 2021TABBIE: Pretrained Representations of Tabular Datacited by 111
  2. 2021Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Basescited by 107
  3. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  4. 2021Representing Numbers in NLP: a Survey and a Visioncited by 83
  5. 2022A Causal Framework to Quantify the Robustness of Mathematical Reasoning with Language Modelscited by 68
  6. 2022Probing for the Usage of Grammatical Numbercited by 57
  7. 2023Generative Table Pre-training Empowers Models for Tabular Predictioncited by 55
  8. 2021CombOptNet: Fit the Right NP-Hard Problem by Learning Integer Programming Constraintscited by 49
  9. 2022Reasoning Like Program Executorscited by 45
  10. 2021Clinical Outcome Prediction from Admission Notes using Self-Supervised Knowledge Integrationcited by 43
  11. 2021A Closer Look at How Fine-tuning Changes BERTcited by 42
  12. 2024Number Cookbook: Number Understanding of Language Models and How to Improve Itcited by 41
  13. 2021Turning Tables: Generating Examples from Semi-structured Tables for Endowing Language Models with Reasoning Skillscited by 39
  14. 2022Commonsense Knowledge Reasoning and Generation with Pre-trained Language Models: A Surveycited by 38
  15. 2021Annotating Columns with Pre-trained Language Modelscited by 37
  16. 2023Mass-Producing Failures of Multimodal Systems with Language Modelscited by 36
  17. 2021Data-to-text Generation with Macro Planningcited by 34
  18. 2021Zero-shot Generalization in Dialog State Tracking through Generative Question Answeringcited by 33
  19. 2025Ignore the KL Penalty! Boosting Exploration on Critical Tokens to Enhance RL Fine-Tuningcited by 33
  20. 2021MWP-BERT: Numeracy-Augmented Pre-training for Math Word Problem Solvingcited by 32
  21. 2024Language Models Encode Numbers Using Digit Representations in Base 10cited by 32
  22. 2023LLM-Prop: Predicting Physical And Electronic Properties Of Crystalline Solids From Their Text Descriptionscited by 29
  23. 2026TOPReward: Token Probabilities as Hidden Zero-Shot Rewards for Roboticscited by 29
  24. 2021DirectProbe: Studying Representations without Classifierscited by 25
  25. 2023Mathematical Language Models: A Surveycited by 24
  26. 2023The Life Cycle of Knowledge in Big Language Models: A Surveycited by 23
  27. 2023RELIC: Investigating Large Language Model Responses using Self-Consistencycited by 23
  28. 2024A Philosophical Introduction to Language Models -- Part I: Continuity With Classic Debatescited by 22
  29. 2023Mitigating Temporal Misalignment by Discarding Outdated Factscited by 21
  30. 2023Can Language Models Understand Physical Concepts?cited by 19
  31. 2024Language Models Encode the Value of Numbers Linearlycited by 19
  32. 2023Can neural networks do arithmetic? A survey on the elementary numerical skills of state-of-the-art deep learning modelscited by 18
  33. 2021Exploring Generalization Ability of Pretrained Language Models on Arithmetic and Logical Reasoningcited by 17
  34. 2021Probing Language Models for Understanding of Temporal Expressionscited by 16
  35. 2022NumHTML: Numeric-Oriented Hierarchical Transformer Model for Multi-task Financial Forecastingcited by 16
  36. 2021NumGPT: Improving Numeracy Ability of Generative Pre-trained Modelscited by 15
  37. 2023Evaluating Transformer Language Models on Arithmetic Operations Using Number Decompositioncited by 15
  38. 2024Semantic Sensitivities and Inconsistent Predictions: Measuring the Fragility of NLI Modelscited by 14

50 most cited of 157 All 157 (JSON)