mlsanityData to 14 Oct 2026

A Survey of Knowledge Enhanced Pre-trained Models

arXiv 2110.00269 ↗First posted 1 Oct 2021cs.CL cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

Pre-trained language models learn informative word representations on a large-scale text corpus through self-supervised learning, which has achieved promising performance in fields of natural language processing (NLP) after fine-tuning. These models, however, suffer from poor robustness and lack of interpretability. We refer to pre-trained language models with knowledge injection as knowledge-enhanced pre-trained language models (KEPLMs). These models demonstrate deep understanding and logical reasoning and introduce interpretability. In this survey, we provide a comprehensive overview of KEPLMs in NLP. We first discuss the advancements in pre-trained language models and knowledge representation learning. Then we systematically categorize existing KEPLMs from three different perspectives. Finally, we outline some potential directions of KEPLMs for future research.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 76 of its 121 references that are in the map. First posted on arXiv 1 Oct 2021.

Builds on

76 of 121 references.

References · 76

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  7. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  8. 2017Graph Attention Networkscited by 8,923
  9. 2017Inductive Representation Learning on Large Graphscited by 6,320
  10. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  11. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  12. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
38 more
  1. 2018Deep contextualized word representationscited by 2,290
  2. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  3. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  4. 2019Language Models as Knowledge Bases?cited by 1,936
  5. 2017Modeling Relational Data with Graph Convolutional Networkscited by 1,694
  6. 2016Enriching Word Vectors with Subword Informationcited by 1,583
  7. 2016Variational Graph Auto-Encoderscited by 1,382
  8. 2013Spectral Networks and Locally Connected Networks on Graphscited by 1,102
  9. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  10. 2014Embedding Entities and Relations for Learning and Inference in Knowledge Basescited by 932
  11. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  12. 2017Convolutional 2D Knowledge Graph Embeddingscited by 730
  13. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 719
  14. 2020A Survey on Knowledge Graphs: Representation, Acquisition and Applicationscited by 643
  15. 2019ERNIE: Enhanced Language Representation with Informative Entitiescited by 528
  16. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 505
  17. 2019COMET: Commonsense Transformers for Automatic Knowledge Graph Constructioncited by 450
  18. 2017Zero-Shot Relation Extraction via Reading Comprehensioncited by 426
  19. 2019Composition-based Multi-Relational Graph Convolutional Networkscited by 396
  20. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 372
  21. 2019KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representationcited by 356
  22. 2020LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attentioncited by 326
  23. 2015Skip-Thought Vectorscited by 322
  24. 2020K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapterscited by 319
  25. 2019K-BERT: Enabling Language Representation with Knowledge Graphcited by 304
  26. 2019ERNIE: Enhanced Representation through Knowledge Integrationcited by 302
  27. 2019Abductive Commonsense Reasoningcited by 301
  28. 2019Knowledge Enhanced Contextual Word Representationscited by 296
  29. 2021PTR: Prompt Tuning with Rules for Text Classificationcited by 238
  30. 2019ERNIE 2.0: A Continual Pre-training Framework for Language Understandingcited by 231
  31. 2020The Next Decade in AI: Four Steps Towards Robust Artificial Intelligencecited by 195
  32. 2015Holographic Embeddings of Knowledge Graphscited by 181
  33. 2016FVQA: Fact-based Visual Question Answeringcited by 181
  34. 2019oLMpics -- On what Language Model Pre-training Capturescited by 161
  35. 2018End-to-end Structure-Aware Convolutional Networks for Knowledge Base Completioncited by 136
  36. 2019Learning Attention-based Embeddings for Relation Prediction in Knowledge Graphscited by 125
  37. 2019Utilizing BERT for Aspect-Based Sentiment Analysis via Constructing Auxiliary Sentencecited by 122
  38. 2020A Knowledge-Enhanced Pretraining Model for Commonsense Story Generationcited by 116

50 most cited of 76 All 76 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 0; 2025: 1; 2026: 1.

3 papers.

  1. 2023Model-as-a-Service (MaaS): A Surveycited by 25
  2. 2025Memorizing is Not Enough: Deep Knowledge Injection Through Reasoningcited by 4
  3. 2026Linguistic Signatures for Enhanced Emotion Detectioncited by 1