# Incorporating Commonsense Knowledge Graph in Pretrained Models for Social Commonsense Tasks

arXiv: https://arxiv.org/abs/2105.05457 · HTML: https://mlsanity.com/paper/2105.05457/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 29 references that are in the map.
- First posted on arXiv 12 May 2021.

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016)
- [UnifiedQA: Crossing Format Boundaries With a Single QA System](https://mlsanity.com/paper/2005.00700/) (2020)
- [Explain Yourself! Leveraging Language Models for Commonsense Reasoning](https://mlsanity.com/paper/1906.02361/) (2019)
- [Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning](https://mlsanity.com/paper/1909.00277/) (2019)
- [Bidirectional Attention Flow for Machine Comprehension](https://mlsanity.com/paper/1611.01603/) (2016)
- [Unsupervised Commonsense Question Answering with Self-Talk](https://mlsanity.com/paper/2004.05483/) (2020)
- [Language Generation with Multi-Hop Reasoning on Commonsense Knowledge Graph](https://mlsanity.com/paper/2009.11692/) (2020)
- [How Additional Knowledge can Improve Natural Language Commonsense Question Answering?](https://mlsanity.com/paper/1909.08855/) (2019)
- [Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-shot Commonsense Question Answering](https://mlsanity.com/paper/1911.03876/) (2019)
- [Yuanfudao at SemEval-2018 Task 11: Three-way Attention and Relational Knowledge for Commonsense Machine Comprehension](https://mlsanity.com/paper/1803.00191/) (2018)
- [Generating Commonsense Explanation by Extracting Bridge Concepts from Reasoning Paths](https://mlsanity.com/paper/2009.11753/) (2020)

## Built on by (most cited first, 24 of 24)

- [Editing Large Language Models: Problems, Methods, and Opportunities](https://mlsanity.com/paper/2305.13172/) (2023)
- [Generated Knowledge Prompting for Commonsense Reasoning](https://mlsanity.com/paper/2110.08387/) (2021)
- [Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey](https://mlsanity.com/paper/2210.07700/) (2022)
- [NormBank: A Knowledge Bank of Situational Social Norms](https://mlsanity.com/paper/2305.17008/) (2023)
- [Give Us the Facts: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modeling](https://mlsanity.com/paper/2306.11489/) (2023)
- [Rainier: Reinforced Knowledge Introspector for Commonsense Question Answering](https://mlsanity.com/paper/2210.03078/) (2022)
- [Human Parity on CommonsenseQA: Augmenting Self-Attention with External Attention](https://mlsanity.com/paper/2112.03254/) (2021)
- [Kformer: Knowledge Injection in Transformer Feed-Forward Layers](https://mlsanity.com/paper/2201.05742/) (2022)
- [PeaCoK: Persona Commonsense Knowledge for Consistent and Engaging Narratives](https://mlsanity.com/paper/2305.02364/) (2023)
- [Mind the Gap! Injecting Commonsense Knowledge for Abstractive Dialogue Summarization](https://mlsanity.com/paper/2209.00930/) (2022)
- [Tox-BART: Leveraging Toxicity Attributes for Explanation Generation of Implicit Hate Speech](https://mlsanity.com/paper/2406.03953/) (2024)
- [Elaboration-Generating Commonsense Question Answering at Scale](https://mlsanity.com/paper/2209.01232/) (2022)
- [Text-Augmented Open Knowledge Graph Completion via Pre-Trained Language Models](https://mlsanity.com/paper/2305.15597/) (2023)
- [Knowledge Graph Tuning: Real-time Large Language Model Personalization based on Human Feedback](https://mlsanity.com/paper/2405.19686/) (2024)
- [Modularized Transfer Learning with Multiple Knowledge Graphs for Zero-shot Commonsense Reasoning](https://mlsanity.com/paper/2206.03715/) (2022)
- [A Survey on Knowledge-Enhanced Pre-trained Language Models](https://mlsanity.com/paper/2212.13428/) (2022)
- [Text-To-KG Alignment: Comparing Current Methods on Classification Tasks](https://mlsanity.com/paper/2306.02871/) (2023)
- [Semantic Categorization of Social Knowledge for Commonsense Question Answering](https://mlsanity.com/paper/2109.05168/) (2021)
- [IEKG: A Commonsense Knowledge Graph for Idiomatic Expressions](https://mlsanity.com/paper/2312.06053/) (2023)
- [Rule or Story, Which is a Better Commonsense Expression for Talking with Large Language Models?](https://mlsanity.com/paper/2402.14355/) (2024)
- [Interviewer-Candidate Role Play: Towards Developing Real-World NLP Systems](https://mlsanity.com/paper/2107.00315/) (2021)
- [Open-ended Commonsense Reasoning with Unrestricted Answer Scope](https://mlsanity.com/paper/2310.11672/) (2023)
- [Moral Sparks in Social Media Narratives](https://mlsanity.com/paper/2310.19268/) (2023)
- [Text2Model: Modeling Copilots for Text-to-Model Translation](https://mlsanity.com/paper/2604.12955/) (2026)

## Abstract (arXiv)

Pretrained language models have excelled at many NLP tasks recently; however, their social intelligence is still unsatisfactory. To enable this, machines need to have a more general understanding of our complicated world and develop the ability to perform commonsense reasoning besides fitting the specific downstream tasks. External commonsense knowledge graphs (KGs), such as ConceptNet, provide rich information about words and their relationships. Thus, towards general commonsense learning, we propose two approaches to implicitly and explicitly infuse such KGs into pretrained language models. We demonstrate our proposed methods perform well on SocialIQA, a social commonsense reasoning task, in both limited and full training data regimes.