# Do Language Models Perform Generalizable Commonsense Inference?

arXiv: https://arxiv.org/abs/2106.11533 · HTML: https://mlsanity.com/paper/2106.11533/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 33 references that are in the map.
- First posted on arXiv 22 Jun 2021.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016)
- [COMET: Commonsense Transformers for Automatic Knowledge Graph Construction](https://mlsanity.com/paper/1906.05317/) (2019)
- [COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphs](https://mlsanity.com/paper/2010.05953/) (2020)
- [Commonsense Knowledge Mining from Pretrained Models](https://mlsanity.com/paper/1909.00505/) (2019)
- [Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgetting](https://mlsanity.com/paper/2004.12651/) (2020)
- [Joint Constrained Learning for Event-Event Relation Extraction](https://mlsanity.com/paper/2010.06727/) (2020)
- [Connecting the Dots: A Knowledgeable Path Generator for Commonsense Question Answering](https://mlsanity.com/paper/2005.00691/) (2020)
- [Automated Storytelling via Causal, Commonsense Plot Ordering](https://mlsanity.com/paper/2009.00829/) (2020)
- [Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-shot Commonsense Question Answering](https://mlsanity.com/paper/1911.03876/) (2019)
- [Commonsense mining as knowledge base completion? A study on the impact of novelty](https://mlsanity.com/paper/1804.09259/) (2018)
- [Commonsense Knowledge Graph Reasoning by Selection or Generation? Why?](https://mlsanity.com/paper/2008.05925/) (2020)

## Built on by (most cited first, 10 of 10)

- [ChatGPT is a Knowledgeable but Inexperienced Solver: An Investigation of Commonsense Problem in Large Language Models](https://mlsanity.com/paper/2303.16421/) (2023)
- [LLMParser: An Exploratory Study on Using Large Language Models for Log Parsing](https://mlsanity.com/paper/2404.18001/) (2024)
- [CAT: A Contextualized Conceptualization and Instantiation Framework for Commonsense Reasoning](https://mlsanity.com/paper/2305.04808/) (2023)
- [Acquiring and Modelling Abstract Commonsense Knowledge via Conceptualization](https://mlsanity.com/paper/2206.01532/) (2022)
- [Robust and Explainable Identification of Logical Fallacies in Natural Language Arguments](https://mlsanity.com/paper/2212.07425/) (2022)
- [Run Like a Girl! Sports-Related Gender Bias in Language and Vision](https://mlsanity.com/paper/2305.14468/) (2023)
- [rmC small IS2: A Simplified Commonsense Inference Evaluation for Story Prose](https://mlsanity.com/paper/2202.07880/) (2022)
- [MemoNet: Memorizing All Cross Features' Representations Efficiently via Multi-Hash Codebook Network for CTR Prediction](https://mlsanity.com/paper/2211.01334/) (2022)
- [Complementary Advantages of ChatGPTs and Human Readers in Reasoning: Evidence from English Text Reading Comprehension](https://mlsanity.com/paper/2311.10344/) (2023)
- [Probing Commonsense Knowledge in Pre-trained Language Models with Sense-level Precision and Expanded Vocabulary](https://mlsanity.com/paper/2210.06376/) (2022)

## Abstract (arXiv)

Inspired by evidence that pretrained language models (LMs) encode commonsense knowledge, recent work has applied LMs to automatically populate commonsense knowledge graphs (CKGs). However, there is a lack of understanding on their generalization to multiple CKGs, unseen relations, and novel entities. This paper analyzes the ability of LMs to perform generalizable commonsense inference, in terms of knowledge capacity, transferability, and induction. Our experiments with these three aspects show that: (1) LMs can adapt to different schemas defined by multiple CKGs but fail to reuse the knowledge to generalize to new relations. (2) Adapted LMs generalize well to unseen subjects, but less so on novel objects. Future work should investigate how to improve the transferability and induction of commonsense mining from LMs.