# A Systematic Investigation of Commonsense Knowledge in Large Language Models

arXiv: https://arxiv.org/abs/2111.00607 · HTML: https://mlsanity.com/paper/2111.00607/

## Key facts

- Cited by 56 papers in the map (data through 7 Oct 2026).
- Builds on 39 of its 66 references that are in the map.
- First posted on arXiv 31 Oct 2021.

## Builds on (most cited first, 30 of 39)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [Deep Reinforcement Learning that Matters](https://mlsanity.com/paper/1709.06560/) (2017)
- [Carbon Emissions and Large Neural Network Training](https://mlsanity.com/paper/2104.10350/) (2021)
- [Defending Against Neural Fake News](https://mlsanity.com/paper/1905.12616/) (2019)
- [COMET: Commonsense Transformers for Automatic Knowledge Graph Construction](https://mlsanity.com/paper/1906.05317/) (2019)
- [Gender Bias in Coreference Resolution](https://mlsanity.com/paper/1804.09301/) (2018)
- [How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation](https://mlsanity.com/paper/1603.08023/) (2016)
- [Six Challenges for Neural Machine Translation](https://mlsanity.com/paper/1706.03872/) (2017)
- [SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference](https://mlsanity.com/paper/1808.05326/) (2018)
- [Abductive Commonsense Reasoning](https://mlsanity.com/paper/1908.05739/) (2019)
- [Hypothesis Only Baselines in Natural Language Inference](https://mlsanity.com/paper/1805.01042/) (2018)
- [All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text](https://mlsanity.com/paper/2107.00061/) (2021)
- [Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning](https://mlsanity.com/paper/1909.00277/) (2019)
- [Surface Form Competition: Why the Highest Probability Answer Isn't Always Right](https://mlsanity.com/paper/2104.08315/) (2021)
- [Experience Grounds Language](https://mlsanity.com/paper/2004.10151/) (2020)
- [Why We Need New Evaluation Metrics for NLG](https://mlsanity.com/paper/1707.06875/) (2017)
- [WikiHow: A Large Scale Text Summarization Dataset](https://mlsanity.com/paper/1810.09305/) (2018)
- [Demographic Dialectal Variation in Social Media: A Case Study of African-American English](https://mlsanity.com/paper/1608.08868/) (2016)
- [Commonsense Knowledge Mining from Pretrained Models](https://mlsanity.com/paper/1909.00505/) (2019)
- [Unsupervised Commonsense Question Answering with Self-Talk](https://mlsanity.com/paper/2004.05483/) (2020)
- [On NMT Search Errors and Model Errors: Cat Got Your Tongue?](https://mlsanity.com/paper/1908.10090/) (2019)

## Built on by (most cited first, 30 of 56)

- [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021)
- [Unifying Large Language Models and Knowledge Graphs: A Roadmap](https://mlsanity.com/paper/2306.08302/) (2023)
- [Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts](https://mlsanity.com/paper/2305.13300/) (2023)
- [A Comprehensive Study of Knowledge Editing for Large Language Models](https://mlsanity.com/paper/2401.01286/) (2024)
- [Large Multimodal Agents: A Survey](https://mlsanity.com/paper/2402.15116/) (2024)
- [OMNI: Open-endedness via Models of human Notions of Interestingness](https://mlsanity.com/paper/2306.01711/) (2023)
- [Mind's Eye of LLMs: Visualization-of-Thought Elicits Spatial Reasoning in Large Language Models](https://mlsanity.com/paper/2404.03622/) (2024)
- [ChatGPT is a Knowledgeable but Inexperienced Solver: An Investigation of Commonsense Problem in Large Language Models](https://mlsanity.com/paper/2303.16421/) (2023)
- [Tree-Planner: Efficient Close-loop Task Planning with Large Language Models](https://mlsanity.com/paper/2310.08582/) (2023)
- [When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Models](https://mlsanity.com/paper/2405.10255/) (2024)
- [Turning English-centric LLMs Into Polyglots: How Much Multilinguality Is Needed?](https://mlsanity.com/paper/2312.12683/) (2023)
- [Answer Matching Outperforms Multiple Choice for Language Model Evaluation](https://mlsanity.com/paper/2507.02856/) (2025)
- [Thought Cloning: Learning to Think while Acting by Imitating Human Thinking](https://mlsanity.com/paper/2306.00323/) (2023)
- [HARE: Explainable Hate Speech Detection with Step-by-Step Reasoning](https://mlsanity.com/paper/2311.00321/) (2023)
- [FirePlace: Geometric Refinements of LLM Common Sense Reasoning for 3D Object Placement](https://mlsanity.com/paper/2503.04919/) (2025)
- [Open-world Story Generation with Structured Knowledge Enhancement: A Comprehensive Survey](https://mlsanity.com/paper/2212.04634/) (2022)
- [A Survey of Optimization-based Task and Motion Planning: From Classical To Learning Approaches](https://mlsanity.com/paper/2404.02817/) (2024)
- [A Sentence is Worth a Thousand Pictures: Can Large Language Models Understand Hum4n L4ngu4ge and the W0rld behind W0rds?](https://mlsanity.com/paper/2308.00109/) (2023)
- [Learning to Reason over Scene Graphs: A Case Study of Finetuning GPT-2 into a Robot Language Model for Grounded Task Planning](https://mlsanity.com/paper/2305.07716/) (2023)
- [Gemini in Reasoning: Unveiling Commonsense in Multimodal Large Language Models](https://mlsanity.com/paper/2312.17661/) (2023)
- [In-Context Exemplars as Clues to Retrieving from Large Associative Memory](https://mlsanity.com/paper/2311.03498/) (2023)
- [Can LLMs be Good Financial Advisors?: An Initial Study in Personal Decision Making for Optimized Outcomes](https://mlsanity.com/paper/2307.07422/) (2023)
- [Do LLMs Really Adapt to Domains? An Ontology Learning Perspective](https://mlsanity.com/paper/2407.19998/) (2024)
- [Always Tell Me The Odds: Fine-grained Conditional Probability Estimation](https://mlsanity.com/paper/2505.01595/) (2025)
- [CPopQA: Ranking Cultural Concept Popularity by LLMs](https://mlsanity.com/paper/2311.07897/) (2023)
- [Contextual Emotion Recognition using Large Vision Language Models](https://mlsanity.com/paper/2405.08992/) (2024)
- [Human-AI Safety: A Descendant of Generative AI and Control Systems Safety](https://mlsanity.com/paper/2405.09794/) (2024)
- [GRASP: A Grid-Based Benchmark for Evaluating Commonsense Spatial Reasoning](https://mlsanity.com/paper/2407.01892/) (2024)
- [Multi-Agent Planning Using Visual Language Models](https://mlsanity.com/paper/2408.05478/) (2024)
- [Fighting Bias with Bias: Promoting Model Robustness by Amplifying Dataset Biases](https://mlsanity.com/paper/2305.18917/) (2023)

## Abstract (arXiv)

Language models (LMs) trained on large amounts of data have shown impressive performance on many NLP tasks under the zero-shot and few-shot setup. Here we aim to better understand the extent to which such models learn commonsense knowledge -- a critical component of many NLP applications. We conduct a systematic and rigorous zero-shot and few-shot commonsense evaluation of large pre-trained LMs, where we: (i) carefully control for the LMs' ability to exploit potential surface cues and annotation artefacts, and (ii) account for variations in performance that arise from factors that are not related to commonsense knowledge. Our findings highlight the limitations of pre-trained LMs in acquiring commonsense knowledge without task-specific supervision; furthermore, using larger models or few-shot evaluation are insufficient to achieve human-level commonsense performance.