mlsanityData to 7 Oct 2026

Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-trained Language Models

arXiv 2005.00683 ↗First posted 2 May 2020cs.CL cs.AIcited by 110

Not on the map. Why?

Cited by 21 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent works show that pre-trained language models (PTLMs), such as BERT, possess certain commonsense and factual knowledge. They suggest that it is promising to use PTLMs as "neural knowledge bases" via predicting masked words. Surprisingly, we find that this may not work for numerical commonsense knowledge (e.g., a bird usually has two legs). In this paper, we investigate whether and to what extent we can induce numerical commonsense knowledge from PTLMs as well as the robustness of this process. To study this, we introduce a novel probing task with a diagnostic dataset, NumerSense, containing 13.6k masked-word-prediction probes (10.5k for fine-tuning and 3.1k for testing). Our analysis reveals that: (1) BERT and its stronger variant RoBERTa perform poorly on the diagnostic dataset prior to any fine-tuning; (2) fine-tuning with distant supervision brings some improvement; (3) the best supervised model still performs poorly as compared to human performance (54.06% vs 96.3% in accuracy).

Key facts. Cited by 110 papers in the map (data through 7 Oct 2026). First posted on arXiv 2 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 28; 2022: 35; 2023: 25; 2024: 13; 2025: 6; 2026: 3.

110 papers.

Sort:

By area:

  1. 2021MetaICL: Learning to Learn In Contextcited by 450
  2. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 333
  3. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoningcited by 331
  4. 2021Generated Knowledge Prompting for Commonsense Reasoningcited by 213
  5. 2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 179
  6. 2021CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLPcited by 142
  7. 2022Impact of Pretraining Term Frequencies on Few-Shot Reasoningcited by 142
  8. 2021Investigating the Limitations of Transformers with Simple Arithmetic Taskscited by 136
  9. 2022TransTab: Learning Transferable Tabular Transformers Across Tablescited by 132
  10. 2022A Survey of Deep Learning for Mathematical Reasoningcited by 124
  11. 2022MultiHiertt: Numerical Reasoning over Multi Hierarchical Tabular and Textual Datacited by 120
  12. 2021Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Basescited by 107
38 more
  1. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  2. 2021QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehensioncited by 91
  3. 2021CREAK: A Dataset for Commonsense Reasoning over Entity Knowledgecited by 88
  4. 2021Pre-trained Language Models in Biomedical Domain: A Systematic Surveycited by 87
  5. 2021Representing Numbers in NLP: a Survey and a Visioncited by 83
  6. 2024In-context Learning with Retrieved Demonstrations for Language Models: A Surveycited by 69
  7. 2021Common Sense Beyond English: Evaluating and Improving Multilingual Language Models for Commonsense Reasoningcited by 64
  8. 2023ChatGPT is a Knowledgeable but Inexperienced Solver: An Investigation of Commonsense Problem in Large Language Modelscited by 63
  9. 2023GeoLLM: Extracting Geospatial Knowledge from Large Language Modelscited by 62
  10. 2021RiddleSense: Reasoning about Riddle Questions Featuring Linguistic Creativity and Commonsense Knowledgecited by 60
  11. 2021The Web Is Your Oyster - Knowledge-Intensive NLP against a Very Large Web Corpuscited by 60
  12. 2021TIMEDIAL: Temporal Commonsense Reasoning in Dialogcited by 55
  13. 2022Perturbation Augmentation for Fairer NLPcited by 51
  14. 2021PROST: Physical Reasoning of Objects through Space and Timecited by 48
  15. 2021Language Model as an Annotator: Exploring DialoGPT for Dialogue Summarizationcited by 46
  16. 2023Vera: A General-Purpose Plausibility Estimation Model for Commonsense Statementscited by 44
  17. 2022Rainier: Reinforced Knowledge Introspector for Commonsense Question Answeringcited by 43
  18. 2022Identifying Weaknesses in Machine Translation Metrics Through Minimum Bayes Risk Decoding: A Case Study for COMETcited by 41
  19. 2024Number Cookbook: Number Understanding of Language Models and How to Improve Itcited by 41
  20. 2021COM2SENSE: A Commonsense Reasoning Benchmark with Complementary Sentencescited by 40
  21. 2023Benchmarks for Automated Commonsense Reasoning: A Surveycited by 40
  22. 2022Commonsense Knowledge Reasoning and Generation with Pre-trained Language Models: A Surveycited by 38
  23. 2024IndoCulture: Exploring Geographically-Influenced Cultural Commonsense Reasoning Across Eleven Indonesian Provincescited by 38
  24. 2021The World of an Octopus: How Reporting Bias Influences a Language Model's Perception of Colorcited by 32
  25. 2021Relational World Knowledge Representation in Contextual Language Models: A Reviewcited by 31
  26. 2022GeoMLAMA: Geo-Diverse Commonsense Probing on Multilingual Pre-Trained Language Modelscited by 30
  27. 2021Dimensions of Commonsense Knowledgecited by 29
  28. 2021Exploring Universal Intrinsic Task Subspace via Prompt Tuningcited by 25
  29. 2023The Life Cycle of Knowledge in Big Language Models: A Surveycited by 23
  30. 2023Tailoring Self-Rationalizers with Multi-Reward Distillationcited by 21
  31. 2021What do Large Language Models Learn about Scripts?cited by 19
  32. 2023Can neural networks do arithmetic? A survey on the elementary numerical skills of state-of-the-art deep learning modelscited by 18
  33. 2022"John is 50 years old, can his son be 65?" Evaluating NLP Models' Understanding of Feasibilitycited by 15
  34. 2023Crystal: Introspective Reasoners Reinforced with Self-Feedbackcited by 15
  35. 2024Paraphrase Types Elicit Prompt Engineering Capabilitiescited by 15
  36. 2021Accurate, yet inconsistent? Consistency Analysis on Language Understanding Modelscited by 13
  37. 2021Investigating Numeracy Learning Ability of a Text-to-Text Transfer Modelcited by 13
  38. 2022Can Pre-trained Language Models Interpret Similes as Smart as Human?cited by 13

50 most cited of 110 All 110 (JSON)