mlsanityData to 7 Oct 2026

Does Knowledge Help General NLU? An Empirical Study

arXiv 2109.00563 ↗First posted 1 Sep 2021cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

It is often observed in knowledge-centric tasks (e.g., common sense question and answering, relation classification) that the integration of external knowledge such as entity representation into language models can help provide useful information to boost the performance. However, it is still unclear whether this benefit can extend to general natural language understanding (NLU) tasks. In this work, we empirically investigated the contribution of external knowledge by measuring the end-to-end performance of language models with various knowledge integration methods. We find that the introduction of knowledge can significantly improve the results on certain tasks while having no adverse effects on other tasks. We then employ mutual information to reflect the difference brought by knowledge and a neural interpretation model to reveal how a language model utilizes external knowledge. Our study provides valuable insights and guidance for practitioners to equip NLP models with knowledge.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 25 of its 37 references that are in the map. First posted on arXiv 1 Sep 2021.

Builds on

25 of 37 references.

References · 25

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  6. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  7. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  8. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  9. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  10. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  11. 2018Neural Network Acceptability Judgmentscited by 952
  12. 2019ERNIE: Enhanced Language Representation with Informative Entitiescited by 525
13 more
  1. 2020LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attentioncited by 326
  2. 2020K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapterscited by 318
  3. 2019K-BERT: Enabling Language Representation with Knowledge Graphcited by 303
  4. 2019ERNIE: Enhanced Representation through Knowledge Integrationcited by 301
  5. 2019Knowledge Enhanced Contextual Word Representationscited by 294
  6. 2018FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluationcited by 254
  7. 2019Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Modelcited by 102
  8. 2019E-BERT: Efficient-Yet-Effective Entity Embeddings for BERTcited by 96
  9. 2018Ultra-Fine Entity Typingcited by 87
  10. 2020JAKET: Joint Pre-training of Knowledge Graph and Language Understandingcited by 73
  11. 2021Revisiting Locally Supervised Learning: an Alternative to End-to-end Trainingcited by 71
  12. 2020How do Decisions Emerge across Layers in Neural Models? Interpretation with Differentiable Maskingcited by 63
  13. 2020REL: An Entity Linker Standing on the Shoulders of Giantscited by 61

Built on by

Cited by year

Citing papers per year: 2021: 3; 2022: 4.

7 papers.

  1. 2021KG-FiD: Infusing Knowledge Graph in Fusion-in-Decoder for Open-Domain Question Answeringcited by 66
  2. 2021Dict-BERT: Enhancing Language Model Pre-training with Dictionarycited by 38
  3. 2022Leveraging Visual Knowledge in Language Tasks: An Empirical Study on Intermediate Pre-training for Cross-modal Knowledge Transfercited by 10
  4. 2021Leveraging Knowledge in Multilingual Commonsense Reasoningcited by 9
  5. 2022REKnow: Enhanced Knowledge for Joint Entity and Relation Extractioncited by 3
  6. 2022An Empirical Revisiting of Linguistic Knowledge Fusion in Language Understanding Taskscited by 2
  7. 2022Ered: Enhanced Text Representations with Entities and Descriptionscited by 1