# Great Truths are Always Simple: A Rather Simple Knowledge Encoder for Enhancing the Commonsense Reasoning Capacity of Pre-Trained Models

arXiv: https://arxiv.org/abs/2205.01841 · HTML: https://mlsanity.com/paper/2205.01841/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 29 of its 40 references that are in the map.
- First posted on arXiv 4 May 2022.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 29 of 29)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [Modeling Relational Data with Graph Convolutional Networks](https://mlsanity.com/paper/1703.06103/) (2017)
- [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016)
- [On the Variance of the Adaptive Learning Rate and Beyond](https://mlsanity.com/paper/1908.03265/) (2019)
- [ERNIE: Enhanced Language Representation with Informative Entities](https://mlsanity.com/paper/1905.07129/) (2019)
- [QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering](https://mlsanity.com/paper/2104.06378/) (2021)
- [KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation](https://mlsanity.com/paper/1911.06136/) (2019)
- [A simple neural network module for relational reasoning](https://mlsanity.com/paper/1706.01427/) (2017)
- [KagNet: Knowledge-Aware Graph Networks for Commonsense Reasoning](https://mlsanity.com/paper/1909.02151/) (2019)
- [Enhanced LSTM for Natural Language Inference](https://mlsanity.com/paper/1609.06038/) (2016)
- [Unsupervised Commonsense Question Answering with Self-Talk](https://mlsanity.com/paper/2004.05483/) (2020)
- [Scalable Multi-Hop Relational Reasoning for Knowledge-Aware Question Answering](https://mlsanity.com/paper/2005.00646/) (2020)
- [CommonsenseQA 2.0: Exposing the Limits of AI through Gamification](https://mlsanity.com/paper/2201.05320/) (2022)
- [Multi-Hop Knowledge Graph Reasoning with Reward Shaping](https://mlsanity.com/paper/1808.10568/) (2018)
- [CoLAKE: Contextualized Language and Knowledge Embedding](https://mlsanity.com/paper/2010.00309/) (2020)
- [A Survey on Complex Knowledge Base Question Answering: Methods, Challenges and Solutions](https://mlsanity.com/paper/2105.11644/) (2021)
- [From 'F' to 'A' on the N.Y. Regents Science Exams: An Overview of the Aristo Project](https://mlsanity.com/paper/1909.01958/) (2019)
- [Lawyers are Dishonest? Quantifying Representational Harms in Commonsense Knowledge Resources](https://mlsanity.com/paper/2103.11320/) (2021)
- [GNN is a Counter? Revisiting GNN for Question Answering](https://mlsanity.com/paper/2110.03192/) (2021)
- [Refined Commonsense Knowledge from Large-Scale Web Contents](https://mlsanity.com/paper/2112.04596/) (2021)
- [Deep Learning for Answer Sentence Selection](https://mlsanity.com/paper/1412.1632/) (2014)
- [A Semantic-based Method for Unsupervised Commonsense Question Answering](https://mlsanity.com/paper/2105.14781/) (2021)
- [Mining Implicit Entity Preference from User-Item Interaction Data for Knowledge Graph Completion via Adversarial Learning](https://mlsanity.com/paper/2003.12718/) (2020)

## Built on by (most cited first, 9 of 9)

- [StructGPT: A General Framework for Large Language Model to Reason over Structured Data](https://mlsanity.com/paper/2305.09645/) (2023)
- [UniKGQA: Unified Retrieval and Reasoning for Solving Multi-hop Question Answering Over Knowledge Graph](https://mlsanity.com/paper/2212.00959/) (2022)
- [Graph Reasoning for Question Answering with Triplet Retrieval](https://mlsanity.com/paper/2305.18742/) (2023)
- [Preserving Commonsense Knowledge from Pre-trained Language Models via Causal Inference](https://mlsanity.com/paper/2306.10790/) (2023)
- [Open-Set Knowledge-Based Visual Question Answering with Inference Paths](https://mlsanity.com/paper/2310.08148/) (2023)
- [RelBERT: Embedding Relations with Language Models](https://mlsanity.com/paper/2310.00299/) (2023)
- [A Context-Enhanced Framework for Sequential Graph Reasoning](https://mlsanity.com/paper/2412.09056/) (2024)
- [Matching Tasks to Objectives: Fine-Tuning and Prompt-Tuning Strategies for Encoder-Decoder Pre-trained Language Models](https://mlsanity.com/paper/2606.24841/) (2026)
- [Solving Hard Analogy Questions with Relation Embedding Chains](https://mlsanity.com/paper/2310.12379/) (2023)

## Abstract (arXiv)

Commonsense reasoning in natural language is a desired ability of artificial intelligent systems. For solving complex commonsense reasoning tasks, a typical solution is to enhance pre-trained language models~(PTMs) with a knowledge-aware graph neural network~(GNN) encoder that models a commonsense knowledge graph~(CSKG). Despite the effectiveness, these approaches are built on heavy architectures, and can't clearly explain how external knowledge resources improve the reasoning capacity of PTMs. Considering this issue, we conduct a deep empirical analysis, and find that it is indeed relation features from CSKGs (but not node features) that mainly contribute to the performance improvement of PTMs. Based on this finding, we design a simple MLP-based knowledge encoder that utilizes statistical relation paths as features. Extensive experiments conducted on five benchmarks demonstrate the effectiveness of our approach, which also largely reduces the parameters for encoding CSKGs. Our codes and data are publicly available at https://github.com/RUCAIBox/SAFE.