mlsanityData to 7 Oct 2026

A Survey of Knowledge Enhanced Pre-trained Language Models

arXiv 2211.05994 ↗First posted 11 Nov 2022cs.CLcited by 76

Not on the map. Why?

Abstract (arXiv)

Pre-trained Language Models (PLMs) which are trained on large text corpus via self-supervised learning method, have yielded promising performance on various tasks in Natural Language Processing (NLP). However, though PLMs with huge parameters can effectively possess rich knowledge learned from massive training text and benefit downstream tasks at the fine-tuning stage, they still have some limitations such as poor reasoning ability due to the lack of external knowledge. Research has been dedicated to incorporating knowledge into PLMs to tackle these issues. In this paper, we present a comprehensive review of Knowledge Enhanced Pre-trained Language Models (KE-PLMs) to provide a clear insight into this thriving field. We introduce appropriate taxonomies respectively for Natural Language Understanding (NLU) and Natural Language Generation (NLG) to highlight these two main tasks of NLP. For NLU, we divide the types of knowledge into four categories: linguistic knowledge, text knowledge, knowledge graph (KG), and rule knowledge. The KE-PLMs for NLG are categorized into KG-based and retrieval-based methods. Finally, we point out some promising future directions of KE-PLMs.

Key facts. Cited by 76 papers in the map (data through 7 Oct 2026). Builds on 110 of its 145 references that are in the map. First posted on arXiv 11 Nov 2022.

Builds on

110 of 145 references.

References · 110

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  9. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  10. 2022Emergent Abilities of Large Language Modelscited by 2,483
  11. 2018Deep contextualized word representationscited by 2,285
  12. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
38 more
  1. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  2. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  3. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  4. 2019Language Models as Knowledge Bases?cited by 1,922
  5. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  6. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  7. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  8. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  9. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  10. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  11. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  12. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  13. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  14. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  15. 2020Self-supervised Learning: Generative or Contrastivecited by 693
  16. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  17. 2020Knowledge Graphscited by 619
  18. 2019Three scenarios for continual learningcited by 610
  19. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  20. 2018Wizard of Wikipedia: Knowledge-Powered Conversational agentscited by 530
  21. 2019ERNIE: Enhanced Language Representation with Informative Entitiescited by 525
  22. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 413
  23. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 382
  24. 2017Measuring Catastrophic Forgetting in Neural Networkscited by 359
  25. 2019KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representationcited by 356
  26. 2020LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attentioncited by 326
  27. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  28. 2020K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapterscited by 318
  29. 2019K-BERT: Enabling Language Representation with Knowledge Graphcited by 303
  30. 2019ERNIE: Enhanced Representation through Knowledge Integrationcited by 301
  31. 2019Abductive Commonsense Reasoningcited by 300
  32. 2019Knowledge Enhanced Contextual Word Representationscited by 294
  33. 2018Analysis Methods in Neural Language Processing: A Surveycited by 276
  34. 2021ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 258
  35. 2021PTR: Prompt Tuning with Rules for Text Classificationcited by 238
  36. 2021PPT: Pre-trained Prompt Tuning for Few-shot Learningcited by 232
  37. 2022UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Modelscited by 231
  38. 2020ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graphcited by 211

50 most cited of 110 All 110 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 17; 2024: 31; 2025: 19; 2026: 8.

76 papers.

Sort:

By area:

  1. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 663
  2. 2023Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graphcited by 235
  3. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 179
  4. 2023Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMscited by 175
  5. 2023Head-to-Tail: How Knowledgeable are Large Language Models (LLMs)? A.K.A. Will LLMs Replace Knowledge Graphs?cited by 167
  6. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 162
  7. 2024A Survey of Attacks on Large Vision-Language Models: Resources, Advances, and Future Trendscited by 82
  8. 2024Demystifying Chains, Trees, and Graphs of Thoughtscited by 63
  9. 2023Give Us the Facts: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modelingcited by 56
  10. 2023Journey to the Center of the Knowledge Neurons: Discoveries of Language-Independent Knowledge Neurons and Degenerate Knowledge Neuronscited by 56
  11. 2025Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understandingcited by 44
  12. 2023LatEval: An Interactive LLMs Evaluation Benchmark with Incomplete Information from Lateral Thinking Puzzlescited by 41
38 more
  1. 2023Generalized Graph Prompt: Toward a Unification of Pre-Training and Downstream Tasks on Graphscited by 39
  2. 2024FiDeLiS: Faithful Reasoning in Large Language Model for Knowledge Graph Question Answeringcited by 38
  3. 2024Decoding by Contrasting Knowledge: Enhancing LLMs' Confidence on Edited Factscited by 36
  4. 2023How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advancescited by 33
  5. 2024Graph Machine Learning in the Era of Large Language Models (LLMs)cited by 33
  6. 2024Large Language Models for Education: A Surveycited by 32
  7. 2023KnowGPT: Knowledge Graph based Prompting for Large Language Modelscited by 30
  8. 2023DialogueLLM: Context and Emotion Knowledge-Tuned Large Language Models for Emotion Recognition in Conversationscited by 29
  9. 2024KnowPhish: Large Language Models Meet Multimodal Knowledge Graphs for Enhancing Reference-Based Phishing Detectioncited by 29
  10. 2024A Systematic Review on Long-Tailed Learningcited by 27
  11. 2024Can Knowledge Graphs Make Large Language Models More Trustworthy? An Empirical Study Over Open-ended Question Answeringcited by 20
  12. 2024An Upload-Efficient Scheme for Transferring Knowledge From a Server-Side Pre-trained Generator to Clients in Heterogeneous Federated Learningcited by 17
  13. 2024Combining Knowledge Graphs and Large Language Modelscited by 16
  14. 2025Large Language Model (LLM) for Software Security: Code Analysis, Malware Analysis, Reverse Engineeringcited by 15
  15. 2024Research Trends for the Interplay between Large Language Models and Knowledge Graphscited by 14
  16. 2024LLM-Powered Explanations: Unraveling Recommendations Through Subgraph Reasoningcited by 10
  17. 2024Knowledge-aware Text-Image Retrieval for Remote Sensing Imagescited by 9
  18. 2024Evaluating the External and Parametric Knowledge Fusion of Large Language Modelscited by 9
  19. 2024GenAI-powered Multi-Agent Paradigm for Smart Urban Mobility: Opportunities and Challenges for Integrating Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) with Intelligent Transportation Systemscited by 9
  20. 2025IDEAL: Data Equilibrium Adaptation for Multi-Capability Language Model Alignmentcited by 8
  21. 2023BioLORD-2023: Semantic Textual Representations Fusing LLM and Clinical Knowledge Graph Insightscited by 7
  22. 2024Safety Control of Service Robots with LLMs and Embodied Knowledge Graphscited by 7
  23. 2025A Survey on fMRI-based Brain Decoding for Reconstructing Multimodal Stimulicited by 6
  24. 2025Dynamic Compressing Prompts for Efficient Inference of Large Language Modelscited by 6
  25. 2024Enhancing Collaborative Semantics of Language Model-Driven Recommendations via Graph-Aware Learningcited by 5
  26. 2022Graph Learning and Its Advancements on Large Language Models: A Holistic Surveycited by 4
  27. 2025Knowledge Graphs for Enhancing Large Language Models in Entity Disambiguationcited by 4
  28. 2026Beyond Static Models: An Evolving Framework for Continual Learning in Large Language Models across Training Stagescited by 4
  29. 2024Online Training of Large Language Models: Learn while chattingcited by 3
  30. 2024Towards Ontology-Enhanced Representation Learning for Large Language Modelscited by 3
  31. 2024GPT-Powered Elicitation Interview Script Generator for Requirements Engineering Trainingcited by 3
  32. 2025ConTextual: Improving Clinical Text Summarization in LLMs with Context-preserving Token Filtering and Knowledge Graphscited by 3
  33. 2025From Symbolic to Neural and Back: Exploring Knowledge Graph-Large Language Model Synergiescited by 3
  34. 2025SignAligner: Harmonizing Complementary Pose Modalities for Coherent Sign Language Generationcited by 3
  35. 2024Residual Learning and Context Encoding for Adaptive Offline-to-Online Reinforcement Learningcited by 2
  36. 2025Agentic Satellite-Augmented Low-Altitude Economy and Terrestrial Networks: A Survey on Generative Approachescited by 2
  37. 2025DSRAG: A Domain-Specific Retrieval Framework Based on Document-derived Multimodal Knowledge Graphcited by 2
  38. 2026<SOG_k>: One LLM Token for Explicit Graph Structural Understandingcited by 2

50 most cited of 76 All 76 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022A Survey on Knowledge-Enhanced Pre-trained Language Models
  2. 2022KMIR: A Benchmark for Evaluating Knowledge Memorization, Identification and Reasoning Abilities of Language Models
  3. 2023Knowledge Rumination for Pre-trained Language Models
  4. 2025Semantic Mastery: Enhancing LLMs with Advanced Natural Language Understanding
  5. 2023Knowledge Editing for Large Language Models: A Survey
  6. 2022Commonsense Knowledge Reasoning and Generation with Pre-trained Language Models: A Survey
  7. 2025Optimising Language Models for Downstream Tasks: A Post-Training Perspective
  8. 2026KoRe: Compact Knowledge Representations for Large Language Models