mlsanityData to 14 Oct 2026

Knowledge Enhanced Pretrained Language Models: A Compreshensive Survey

arXiv 2110.08455 ↗First posted 16 Oct 2021cs.CLcited by 33

Not on the map. Why?

Abstract (arXiv)

Pretrained Language Models (PLM) have established a new paradigm through learning informative contextualized representations on large-scale text corpus. This new paradigm has revolutionized the entire field of natural language processing, and set the new state-of-the-art performance for a wide variety of NLP tasks. However, though PLMs could store certain knowledge/facts from training corpus, their knowledge awareness is still far from satisfactory. To address this issue, integrating knowledge into PLMs have recently become a very active research area and a variety of approaches have been developed. In this paper, we provide a comprehensive survey of the literature on this emerging and fast-growing field - Knowledge Enhanced Pretrained Language Models (KE-PLMs). We introduce three taxonomies to categorize existing work. Besides, we also survey the various NLU and NLG applications on which KE-PLM has demonstrated superior performance over vanilla PLMs. Finally, we discuss challenges that face KE-PLMs and also promising directions for future research.

Key facts. Cited by 33 papers in the map (data through 14 Oct 2026). Builds on 84 of its 111 references that are in the map. First posted on arXiv 16 Oct 2021.

Builds on

84 of 111 references.

References · 84

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  7. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  8. 2017Graph Attention Networkscited by 8,923
  9. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  10. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  11. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  12. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
38 more
  1. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  2. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  3. 2019Language Models as Knowledge Bases?cited by 1,936
  4. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  5. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  6. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,067
  7. 2019Strategies for Pre-training Graph Neural Networkscited by 971
  8. 2019BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformercited by 964
  9. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 876
  10. 2017Convolutional 2D Knowledge Graph Embeddingscited by 730
  11. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 719
  12. 2018Universal Transformerscited by 647
  13. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 584
  14. 2019ERNIE: Enhanced Language Representation with Informative Entitiescited by 528
  15. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 505
  16. 2019COMET: Commonsense Transformers for Automatic Knowledge Graph Constructioncited by 450
  17. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 417
  18. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 383
  19. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 359
  20. 2019KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representationcited by 356
  21. 2020LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attentioncited by 326
  22. 2019KG-BERT: BERT for Knowledge Graph Completioncited by 324
  23. 2020K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapterscited by 319
  24. 2019K-BERT: Enabling Language Representation with Knowledge Graphcited by 304
  25. 2020GPT-GNN: Generative Pre-Training of Graph Neural Networkscited by 303
  26. 2019Knowledge Enhanced Contextual Word Representationscited by 296
  27. 2019Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoningcited by 285
  28. 2018FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluationcited by 254
  29. 2019Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Flycited by 243
  30. 2019ERNIE 2.0: A Continual Pre-training Framework for Language Understandingcited by 231
  31. 2017SearchQA: A New Q&A Dataset Augmented with Context from a Search Enginecited by 225
  32. 2019KagNet: Knowledge-Aware Graph Networks for Commonsense Reasoningcited by 216
  33. 2019Probing Neural Network Comprehension of Natural Language Argumentscited by 214
  34. 2020How Context Affects Language Models' Factual Predictionscited by 181
  35. 2019oLMpics -- On what Language Model Pre-training Capturescited by 161
  36. 2020Probing Pretrained Language Models for Lexical Semanticscited by 156
  37. 2017Know-Evolve: Deep Temporal Reasoning for Dynamic Knowledge Graphscited by 149
  38. 2020A Knowledge-Enhanced Pretraining Model for Commonsense Story Generationcited by 116

50 most cited of 84 All 84 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 10; 2023: 10; 2024: 9; 2025: 4.

33 papers.

Sort:

By area:

  1. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 668
  2. 2023StructGPT: A General Framework for Large Language Model to Reason over Structured Datacited by 343
  3. 2022A Review on Language Models as Knowledge Basescited by 157
  4. 2023Can Knowledge Graphs Reduce Hallucinations in LLMs? : A Surveycited by 92
  5. 2022A Survey of Knowledge Enhanced Pre-trained Language Modelscited by 76
  6. 2024Demystifying Chains, Trees, and Graphs of Thoughtscited by 63
  7. 2023Give Us the Facts: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modelingcited by 56
  8. 2023How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advancescited by 33
  9. 2022A Survey of Knowledge-Intensive NLP with Pre-Trained Language Modelscited by 30
  10. 2022SpaBERT: A Pretrained Language Model from Geographic Data for Geo-Entity Representationcited by 23
  11. 2023The Life Cycle of Knowledge in Big Language Models: A Surveycited by 23
  12. 2022KCD: Knowledge Walks and Textual Cues Enhanced Political Perspective Detection in News Mediacited by 20
21 more
  1. 2022Visually-Augmented Language Modelingcited by 18
  2. 2024Combining Knowledge Graphs and Large Language Modelscited by 16
  3. 2024Adaptive Question Answering: Enhancing Language Model Proficiency for Addressing Knowledge Conflicts with Source Citationscited by 15
  4. 2025Knowledge Graph-based Retrieval-Augmented Generation for Schema Matchingcited by 13
  5. 2024Knowledge-aware Text-Image Retrieval for Remote Sensing Imagescited by 9
  6. 2024Incorporating External Knowledge and Goal Guidance for LLM-based Conversational Recommender Systemscited by 8
  7. 2025A Survey of Inductive Reasoning for Large Language Modelscited by 8
  8. 2024Knowledge-Guided Dynamic Modality Attention Fusion Framework for Multimodal Sentiment Analysiscited by 7
  9. 2023Distinguish Before Answer: Generating Contrastive Explanation as Knowledge for Commonsense Question Answeringcited by 6
  10. 2023ConKI: Contrastive Knowledge Injection for Multimodal Sentiment Analysiscited by 4
  11. 2025ConfRAG: Confidence-Guided Retrieval-Augmenting Generationcited by 4
  12. 2022Enhancing Document-level Relation Extraction by Entity Knowledge Injectioncited by 3
  13. 2022A Survey on Knowledge-Enhanced Pre-trained Language Modelscited by 3
  14. 2024Foundation Model Sherpas: Guiding Foundation Models through Knowledge and Reasoningcited by 3
  15. 2023Knowledge Graphs are not Created Equal: Exploring the Properties and Structure of Real KGscited by 1
  16. 2023Diversifying Knowledge Enhancement of Biomedical Language Models using Adapter Modules and Knowledge Graphscited by 1
  17. 2024Human Centered AI for Indian Legal Text Analyticscited by 1
  18. 2022KESA: A Knowledge Enhanced Approach For Sentiment Analysiscited by 0
  19. 2022DSTEA: Improving Dialogue State Tracking via Entity Adaptive Pre-trainingcited by 0
  20. 2024Adapter-based Approaches to Knowledge-enhanced Language Models -- A Surveycited by 0
  21. 2025MIDG: Mixture of Invariant Experts with knowledge injection for Domain Generalization in Multimodal Sentiment Analysiscited by 0