# Does Knowledge Help General NLU? An Empirical Study

arXiv: https://arxiv.org/abs/2109.00563 · HTML: https://mlsanity.com/paper/2109.00563/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 37 references that are in the map.
- First posted on arXiv 1 Sep 2021.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [Neural Network Acceptability Judgments](https://mlsanity.com/paper/1805.12471/) (2018)
- [ERNIE: Enhanced Language Representation with Informative Entities](https://mlsanity.com/paper/1905.07129/) (2019)
- [LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention](https://mlsanity.com/paper/2010.01057/) (2020)
- [K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters](https://mlsanity.com/paper/2002.01808/) (2020)
- [K-BERT: Enabling Language Representation with Knowledge Graph](https://mlsanity.com/paper/1909.07606/) (2019)
- [ERNIE: Enhanced Representation through Knowledge Integration](https://mlsanity.com/paper/1904.09223/) (2019)
- [Knowledge Enhanced Contextual Word Representations](https://mlsanity.com/paper/1909.04164/) (2019)
- [FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluation](https://mlsanity.com/paper/1810.10147/) (2018)
- [Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model](https://mlsanity.com/paper/1912.09637/) (2019)
- [E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT](https://mlsanity.com/paper/1911.03681/) (2019)
- [Ultra-Fine Entity Typing](https://mlsanity.com/paper/1807.04905/) (2018)
- [JAKET: Joint Pre-training of Knowledge Graph and Language Understanding](https://mlsanity.com/paper/2010.00796/) (2020)
- [Revisiting Locally Supervised Learning: an Alternative to End-to-end Training](https://mlsanity.com/paper/2101.10832/) (2021)
- [How do Decisions Emerge across Layers in Neural Models? Interpretation with Differentiable Masking](https://mlsanity.com/paper/2004.14992/) (2020)
- [REL: An Entity Linker Standing on the Shoulders of Giants](https://mlsanity.com/paper/2006.01969/) (2020)

## Built on by (most cited first, 7 of 7)

- [KG-FiD: Infusing Knowledge Graph in Fusion-in-Decoder for Open-Domain Question Answering](https://mlsanity.com/paper/2110.04330/) (2021)
- [Dict-BERT: Enhancing Language Model Pre-training with Dictionary](https://mlsanity.com/paper/2110.06490/) (2021)
- [Leveraging Visual Knowledge in Language Tasks: An Empirical Study on Intermediate Pre-training for Cross-modal Knowledge Transfer](https://mlsanity.com/paper/2203.07519/) (2022)
- [Leveraging Knowledge in Multilingual Commonsense Reasoning](https://mlsanity.com/paper/2110.08462/) (2021)
- [REKnow: Enhanced Knowledge for Joint Entity and Relation Extraction](https://mlsanity.com/paper/2206.05123/) (2022)
- [An Empirical Revisiting of Linguistic Knowledge Fusion in Language Understanding Tasks](https://mlsanity.com/paper/2210.13002/) (2022)
- [Ered: Enhanced Text Representations with Entities and Descriptions](https://mlsanity.com/paper/2208.08954/) (2022)

## Abstract (arXiv)

It is often observed in knowledge-centric tasks (e.g., common sense question and answering, relation classification) that the integration of external knowledge such as entity representation into language models can help provide useful information to boost the performance. However, it is still unclear whether this benefit can extend to general natural language understanding (NLU) tasks. In this work, we empirically investigated the contribution of external knowledge by measuring the end-to-end performance of language models with various knowledge integration methods. We find that the introduction of knowledge can significantly improve the results on certain tasks while having no adverse effects on other tasks. We then employ mutual information to reflect the difference brought by knowledge and a neural interpretation model to reveal how a language model utilizes external knowledge. Our study provides valuable insights and guidance for practitioners to equip NLP models with knowledge.