mlsanityData to 7 Oct 2026

A Systematic Investigation of Commonsense Knowledge in Large Language Models

arXiv 2111.00607 ↗First posted 31 Oct 2021cs.CLcited by 56

Not on the map. Why?

Abstract (arXiv)

Language models (LMs) trained on large amounts of data have shown impressive performance on many NLP tasks under the zero-shot and few-shot setup. Here we aim to better understand the extent to which such models learn commonsense knowledge -- a critical component of many NLP applications. We conduct a systematic and rigorous zero-shot and few-shot commonsense evaluation of large pre-trained LMs, where we: (i) carefully control for the LMs' ability to exploit potential surface cues and annotation artefacts, and (ii) account for variations in performance that arise from factors that are not related to commonsense knowledge. Our findings highlight the limitations of pre-trained LMs in acquiring commonsense knowledge without task-specific supervision; furthermore, using larger models or few-shot evaluation are insufficient to achieve human-level commonsense performance.

Key facts. Cited by 56 papers in the map (data through 7 Oct 2026). Builds on 39 of its 66 references that are in the map. First posted on arXiv 31 Oct 2021.

Builds on

39 of 66 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  5. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  7. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  8. 2019Language Models as Knowledge Bases?cited by 1,922
  9. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  10. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  11. 2017Deep Reinforcement Learning that Matterscited by 727
  12. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
27 more
  1. 2019Defending Against Neural Fake Newscited by 553
  2. 2019COMET: Commonsense Transformers for Automatic Knowledge Graph Constructioncited by 450
  3. 2018Gender Bias in Coreference Resolutioncited by 438
  4. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
  5. 2017Six Challenges for Neural Machine Translationcited by 334
  6. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  7. 2019Abductive Commonsense Reasoningcited by 300
  8. 2018Hypothesis Only Baselines in Natural Language Inferencecited by 291
  9. 2021All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Textcited by 287
  10. 2019Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoningcited by 281
  11. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 256
  12. 2020Experience Grounds Languagecited by 241
  13. 2017Why We Need New Evaluation Metrics for NLGcited by 232
  14. 2018WikiHow: A Large Scale Text Summarization Datasetcited by 203
  15. 2016Demographic Dialectal Variation in Social Media: A Case Study of African-American Englishcited by 194
  16. 2019Commonsense Knowledge Mining from Pretrained Modelscited by 189
  17. 2020Unsupervised Commonsense Question Answering with Self-Talkcited by 161
  18. 2019On NMT Search Errors and Model Errors: Cat Got Your Tongue?cited by 114
  19. 2017On the State of the Art of Evaluation in Neural Language Modelscited by 103
  20. 2021UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmarkcited by 98
  21. 2019Evaluating Commonsense in Pre-trained Language Modelscited by 95
  22. 2019Learning and Evaluating General Linguistic Intelligencecited by 78
  23. 2020ProtoQA: A Question Answering Dataset for Prototypical Common-Sense Reasoningcited by 45
  24. 2020Differentiable Open-Ended Commonsense Reasoningcited by 29
  25. 2018How Reasonable are Common-Sense Reasoning Tasks: A Case-Study on the Winograd Schema Challenge and SWAGcited by 18
  26. 2018Machine Common Sense Concept Papercited by 17
  27. 2021Identify, Align, and Integrate: Matching Knowledge Graphs to Commonsense Reasoning Taskscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 2; 2023: 18; 2024: 19; 2025: 14; 2026: 2.

56 papers.

Sort:

By area:

  1. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  2. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 663
  3. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 333
  4. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 157
  5. 2024Large Multimodal Agents: A Surveycited by 99
  6. 2023OMNI: Open-endedness via Models of human Notions of Interestingnesscited by 66
  7. 2024Mind's Eye of LLMs: Visualization-of-Thought Elicits Spatial Reasoning in Large Language Modelscited by 65
  8. 2023ChatGPT is a Knowledgeable but Inexperienced Solver: An Investigation of Commonsense Problem in Large Language Modelscited by 63
  9. 2023Tree-Planner: Efficient Close-loop Task Planning with Large Language Modelscited by 48
  10. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Modelscited by 41
  11. 2023Turning English-centric LLMs Into Polyglots: How Much Multilinguality Is Needed?cited by 39
  12. 2025Answer Matching Outperforms Multiple Choice for Language Model Evaluationcited by 37
38 more
  1. 2023Thought Cloning: Learning to Think while Acting by Imitating Human Thinkingcited by 30
  2. 2023HARE: Explainable Hate Speech Detection with Step-by-Step Reasoningcited by 30
  3. 2025FirePlace: Geometric Refinements of LLM Common Sense Reasoning for 3D Object Placementcited by 24
  4. 2022Open-world Story Generation with Structured Knowledge Enhancement: A Comprehensive Surveycited by 21
  5. 2024A Survey of Optimization-based Task and Motion Planning: From Classical To Learning Approachescited by 19
  6. 2023A Sentence is Worth a Thousand Pictures: Can Large Language Models Understand Hum4n L4ngu4ge and the W0rld behind W0rds?cited by 17
  7. 2023Learning to Reason over Scene Graphs: A Case Study of Finetuning GPT-2 into a Robot Language Model for Grounded Task Planningcited by 16
  8. 2023Gemini in Reasoning: Unveiling Commonsense in Multimodal Large Language Modelscited by 15
  9. 2023In-Context Exemplars as Clues to Retrieving from Large Associative Memorycited by 13
  10. 2023Can LLMs be Good Financial Advisors?: An Initial Study in Personal Decision Making for Optimized Outcomescited by 10
  11. 2024Do LLMs Really Adapt to Domains? An Ontology Learning Perspectivecited by 10
  12. 2025Always Tell Me The Odds: Fine-grained Conditional Probability Estimationcited by 9
  13. 2023CPopQA: Ranking Cultural Concept Popularity by LLMscited by 8
  14. 2024Contextual Emotion Recognition using Large Vision Language Modelscited by 8
  15. 2024Human-AI Safety: A Descendant of Generative AI and Control Systems Safetycited by 7
  16. 2024GRASP: A Grid-Based Benchmark for Evaluating Commonsense Spatial Reasoningcited by 7
  17. 2024Multi-Agent Planning Using Visual Language Modelscited by 6
  18. 2023Fighting Bias with Bias: Promoting Model Robustness by Amplifying Dataset Biasescited by 5
  19. 2024An Evaluation of Large Language Models in Bioinformatics Researchcited by 5
  20. 2022GPT-Neo for commonsense reasoning -- a theoretical and practical lenscited by 4
  21. 2024Reinforcement Learning for Aligning Large Language Models Agents with Interactive Environments: Quantifying and Mitigating Prompt Overfittingcited by 4
  22. 2025A large-scale evaluation of commonsense knowledge in humans and large language modelscited by 4
  23. 2023Language Models as Knowledge Bases for Visual Word Sense Disambiguationcited by 3
  24. 2023Emotional Theory of Mind: Bridging Fast Visual Processing with Slow Linguistic Reasoningcited by 3
  25. 2024PANGeA: Procedural Artificial Narrative using Generative AI for Turn-Based Video Gamescited by 3
  26. 2024Language Model as Visual Explainercited by 3
  27. 2025It Depends: Resolving Referential Ambiguity in Minimal Contexts with Commonsense Knowledgecited by 3
  28. 2026Common to Whom? Regional Cultural Commonsense and LLM Bias in Indiacited by 3
  29. 2023CommonsenseVIS: Visualizing and Understanding Commonsense Reasoning Capabilities of Natural Language Modelscited by 2
  30. 2024ActionCOMET: A Zero-shot Approach to Learn Image-specific Commonsense Concepts about Actionscited by 2
  31. 2025Shades of Zero: Distinguishing Impossibility from Inconceivabilitycited by 2
  32. 2025WinoWhat: A Parallel Corpus of Paraphrased WinoGrande Sentences with Common Sense Categorizationcited by 2
  33. 2025Breaking Language Barriers: Equitable Performance in Multilingual Language Modelscited by 2
  34. 2024Rule or Story, Which is a Better Commonsense Expression for Talking with Large Language Models?cited by 1
  35. 2024Protecting Privacy in Classifiers by Token Manipulationcited by 1
  36. 2024Forecasting Live Chat Intent from Browsing Historycited by 1
  37. 2025Disentangling Exploration of Large Language Models by Optimal Exploitationcited by 1
  38. 2025Large Language Model Data Generation for Enhanced Intent Recognition in German Speechcited by 1

50 most cited of 56 All 56 (JSON)