mlsanityData to 7 Oct 2026

A Survey on Knowledge-Enhanced Pre-trained Language Models

arXiv 2212.13428 ↗First posted 27 Dec 2022cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

Natural Language Processing (NLP) has been revolutionized by the use of Pre-trained Language Models (PLMs) such as BERT. Despite setting new records in nearly every NLP task, PLMs still face a number of challenges including poor interpretability, weak reasoning capability, and the need for a lot of expensive annotated data when applied to downstream tasks. By integrating external knowledge into PLMs, textitunderlineKnowledge-underlineEnhanced underlinePre-trained underlineLanguage underlineModels (KEPLMs) have the potential to overcome the above-mentioned limitations. In this paper, we examine KEPLMs systematically through a series of studies. Specifically, we outline the common types and different formats of knowledge to be integrated into KEPLMs, detail the existing methods for building and evaluating KEPLMS, present the applications of KEPLMs in downstream tasks, and discuss the future research directions. Researchers will benefit from this survey by gaining a quick and comprehensive overview of the latest developments in this field.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 132 of its 187 references that are in the map. First posted on arXiv 27 Dec 2022.

Builds on

132 of 187 references.

References · 132

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  7. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  9. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  10. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  11. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  12. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
38 more
  1. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  2. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  3. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  4. 2018Deep contextualized word representationscited by 2,285
  5. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  6. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  7. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  8. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  9. 2019Language Models as Knowledge Bases?cited by 1,922
  10. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  11. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  12. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  13. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,065
  14. 2019How Can We Know What Language Models Know?cited by 953
  15. 2017Convolutional 2D Knowledge Graph Embeddingscited by 729
  16. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  17. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 600
  18. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 578
  19. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  20. 2021Editing Factual Knowledge in Language Modelscited by 548
  21. 2018Wizard of Wikipedia: Knowledge-Powered Conversational agentscited by 530
  22. 2019ERNIE: Enhanced Language Representation with Informative Entitiescited by 525
  23. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 504
  24. 2019COMET: Commonsense Transformers for Automatic Knowledge Graph Constructioncited by 450
  25. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 413
  26. 2021QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answeringcited by 386
  27. 2019KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representationcited by 356
  28. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  29. 2020LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attentioncited by 326
  30. 2019KG-BERT: BERT for Knowledge Graph Completioncited by 323
  31. 2020K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapterscited by 318
  32. 2019K-BERT: Enabling Language Representation with Knowledge Graphcited by 303
  33. 2019ERNIE: Enhanced Representation through Knowledge Integrationcited by 301
  34. 2019Knowledge Enhanced Contextual Word Representationscited by 294
  35. 2019Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoningcited by 281
  36. 2021ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 258
  37. 2018FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluationcited by 254
  38. 2020COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphscited by 254

50 most cited of 132 All 132 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 3.

3 papers.

  1. 2024Combining Knowledge Graphs and Large Language Modelscited by 16
  2. 2024Foundation Model Sherpas: Guiding Foundation Models through Knowledge and Reasoningcited by 3
  3. 2024Benchmarking Defeasible Reasoning with Large Language Models -- Initial Experiments and Future Directionscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2022A Survey of Knowledge Enhanced Pre-trained Language Models
  2. 2023Knowledge Rumination for Pre-trained Language Models
  3. 2025Optimising Language Models for Downstream Tasks: A Post-Training Perspective
  4. 2025Semantic Mastery: Enhancing LLMs with Advanced Natural Language Understanding
  5. 2023Give Us the Facts: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modeling
  6. 2022KMIR: A Benchmark for Evaluating Knowledge Memorization, Identification and Reasoning Abilities of Language Models
  7. 2023Knowledge Editing for Large Language Models: A Survey
  8. 2024An Enhanced Prompt-Based LLM Reasoning Scheme via Knowledge Graph-Integrated Collaboration