# Commonsense Knowledge Reasoning and Generation with Pre-trained Language Models: A Survey

arXiv: https://arxiv.org/abs/2201.12438 · HTML: https://mlsanity.com/paper/2201.12438/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 56 of its 84 references that are in the map.
- First posted on arXiv 28 Jan 2022.

## Builds on (most cited first, 30 of 56)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020)
- [From Recognition to Cognition: Visual Commonsense Reasoning](https://mlsanity.com/paper/1811.10830/) (2018)
- [Pre-trained Models for Natural Language Processing: A Survey](https://mlsanity.com/paper/2003.08271/) (2020)
- [COMET: Commonsense Transformers for Automatic Knowledge Graph Construction](https://mlsanity.com/paper/1906.05317/) (2019)
- [Explain Yourself! Leveraging Language Models for Commonsense Reasoning](https://mlsanity.com/paper/1906.02361/) (2019)
- [SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference](https://mlsanity.com/paper/1808.05326/) (2018)
- [XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning](https://mlsanity.com/paper/2005.00333/) (2020)
- [What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models](https://mlsanity.com/paper/1907.13528/) (2019)
- [Pre-Trained Models: Past, Present and Future](https://mlsanity.com/paper/2106.07139/) (2021)
- [Abductive Commonsense Reasoning](https://mlsanity.com/paper/1908.05739/) (2019)
- [Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning](https://mlsanity.com/paper/1909.00277/) (2019)
- [COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphs](https://mlsanity.com/paper/2010.05953/) (2020)
- [Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly](https://mlsanity.com/paper/1911.03343/) (2019)
- [IntelliCode Compose: Code Generation Using Transformer](https://mlsanity.com/paper/2005.08025/) (2020)
- [WinoGrande: An Adversarial Winograd Schema Challenge at Scale](https://mlsanity.com/paper/1907.10641/) (2019)
- [Commonsense Knowledge Mining from Pretrained Models](https://mlsanity.com/paper/1909.00505/) (2019)
- [Adversarial Filters of Dataset Biases](https://mlsanity.com/paper/2002.04108/) (2020)
- [Do NLP Models Know Numbers? Probing Numeracy in Embeddings](https://mlsanity.com/paper/1909.07940/) (2019)
- [Injecting Numerical Reasoning Skills into Language Models](https://mlsanity.com/paper/2004.04487/) (2020)
- [AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing](https://mlsanity.com/paper/2108.05542/) (2021)

## Built on by (most cited first, 30 of 38)

- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [Towards Reasoning in Large Language Models: A Survey](https://mlsanity.com/paper/2212.10403/) (2022)
- [Reasoning with Language Model Prompting: A Survey](https://mlsanity.com/paper/2212.09597/) (2022)
- [Large-scale Multi-Modal Pre-trained Models: A Comprehensive Survey](https://mlsanity.com/paper/2302.10035/) (2023)
- [ChatGPT is a Knowledgeable but Inexperienced Solver: An Investigation of Commonsense Problem in Large Language Models](https://mlsanity.com/paper/2303.16421/) (2023)
- [Large Language Models in Education: Vision and Opportunities](https://mlsanity.com/paper/2311.13160/) (2023)
- [Benchmarks for Automated Commonsense Reasoning: A Survey](https://mlsanity.com/paper/2302.04752/) (2023)
- [Editing Common Sense in Transformers](https://mlsanity.com/paper/2305.14956/) (2023)
- [Through the Lens of Core Competency: Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2308.07902/) (2023)
- [Evaluating the Deductive Competence of Large Language Models](https://mlsanity.com/paper/2309.05452/) (2023)
- [An Overview Of Temporal Commonsense Reasoning and Acquisition](https://mlsanity.com/paper/2308.00002/) (2023)
- [Negation, Coordination, and Quantifiers in Contextualized Language Models](https://mlsanity.com/paper/2209.07836/) (2022)
- [Belief in the Machine: Investigating Epistemological Blind Spots of Language Models](https://mlsanity.com/paper/2410.21195/) (2024)
- [Knowledge-augmented Few-shot Visual Relation Detection](https://mlsanity.com/paper/2303.05342/) (2023)
- [The Odyssey of Commonsense Causality: From Foundational Benchmarks to Cutting-Edge Reasoning](https://mlsanity.com/paper/2406.19307/) (2024)
- [The curse of language biases in remote sensing VQA: the role of spatial attributes, language diversity, and the need for clear evaluation](https://mlsanity.com/paper/2311.16782/) (2023)
- [CASA: Causality-driven Argument Sufficiency Assessment](https://mlsanity.com/paper/2401.05249/) (2024)
- [BUCA: A Binary Classification Approach to Unsupervised Commonsense Question Answering](https://mlsanity.com/paper/2305.15932/) (2023)
- [SituatedGen: Incorporating Geographical and Temporal Contexts into Generative Commonsense Reasoning](https://mlsanity.com/paper/2306.12552/) (2023)
- [Language Models as Knowledge Bases for Visual Word Sense Disambiguation](https://mlsanity.com/paper/2310.01960/) (2023)
- [CORECODE: A Common Sense Annotated Dialogue Dataset with Benchmark Tasks for Chinese Large Language Models](https://mlsanity.com/paper/2312.12853/) (2023)
- [PARADISE: Evaluating Implicit Planning Skills of Language Models with Procedural Warnings and Tips Dataset](https://mlsanity.com/paper/2403.03167/) (2024)
- [Diagnosing Failures in Large Language Models' Answers: Integrating Error Attribution into Evaluation Framework](https://mlsanity.com/paper/2507.08459/) (2025)
- [AgentCoMa: A Compositional Benchmark Mixing Commonsense and Mathematical Reasoning in Real-World Scenarios](https://mlsanity.com/paper/2508.19988/) (2025)
- [The Contribution of Knowledge in Visiolinguistic Learning: A Survey on Tasks and Challenges](https://mlsanity.com/paper/2303.02411/) (2023)
- [A Formalism and Approach for Improving Robustness of Large Language Models Using Risk-Adjusted Confidence Scores](https://mlsanity.com/paper/2310.03283/) (2023)
- [Temporal Validity Change Prediction](https://mlsanity.com/paper/2401.00779/) (2024)
- [Large Language Models Are Also Good Prototypical Commonsense Reasoners](https://mlsanity.com/paper/2309.13165/) (2023)
- [Ontology Revision based on Pre-trained Language Models](https://mlsanity.com/paper/2310.18378/) (2023)
- [IEKG: A Commonsense Knowledge Graph for Idiomatic Expressions](https://mlsanity.com/paper/2312.06053/) (2023)

## Abstract (arXiv)

While commonsense knowledge acquisition and reasoning has traditionally been a core research topic in the knowledge representation and reasoning community, recent years have seen a surge of interest in the natural language processing community in developing pre-trained models and testing their ability to address a variety of newly designed commonsense knowledge reasoning and generation tasks. This paper presents a survey of these tasks, discusses the strengths and weaknesses of state-of-the-art pre-trained models for commonsense reasoning and generation as revealed by these tasks, and reflects on future research directions.