mlsanityData to 7 Oct 2026

A Survey of Knowledge-Intensive NLP with Pre-Trained Language Models

arXiv 2202.08772 ↗First posted 17 Feb 2022cs.CLcited by 30

Not on the map. Why?

Abstract (arXiv)

With the increasing of model capacity brought by pre-trained language models, there emerges boosting needs for more knowledgeable natural language processing (NLP) models with advanced functionalities including providing and making flexible use of encyclopedic and commonsense knowledge. The mere pre-trained language models, however, lack the capacity of handling such knowledge-intensive NLP tasks alone. To address this challenge, large numbers of pre-trained language models augmented with external knowledge sources are proposed and in rapid development. In this paper, we aim to summarize the current progress of pre-trained language model-based knowledge-enhanced models (PLMKEs) by dissecting their three vital elements: knowledge sources, knowledge-intensive NLP tasks, and knowledge fusion methods. Finally, we present the challenges of PLMKEs based on the discussion regarding the three elements and attempt to provide NLP practitioners with potential directions for further research.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Builds on 53 of its 75 references that are in the map. First posted on arXiv 17 Feb 2022.

Builds on

53 of 75 references.

References · 53

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  5. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  6. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  9. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  10. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  11. 2019Language Models as Knowledge Bases?cited by 1,922
  12. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
38 more
  1. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  2. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  3. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,065
  4. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  5. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  6. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  7. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  8. 2019ERNIE: Enhanced Language Representation with Informative Entitiescited by 525
  9. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  10. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 413
  11. 2021QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answeringcited by 386
  12. 2019Abductive Commonsense Reasoningcited by 300
  13. 2019Knowledge Enhanced Contextual Word Representationscited by 294
  14. 2021Time-Aware Language Models as Temporal Knowledge Basescited by 290
  15. 2021ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 258
  16. 2020COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphscited by 254
  17. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  18. 2019KagNet: Knowledge-Aware Graph Networks for Commonsense Reasoningcited by 216
  19. 2021Visually Grounded Reasoning across Languages and Culturescited by 188
  20. 2022GreaseLM: Graph REASoning Enhanced Language Models for Question Answeringcited by 170
  21. 2021SituatedQA: Incorporating Extra-Linguistic Contexts into QAcited by 145
  22. 2021End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answeringcited by 133
  23. 2019"Going on a vacation" takes longer than "Going for a walk": A Study of Temporal Commonsense Understandingcited by 121
  24. 2019Fine-grained Fact Verification with Kernel Graph Attention Networkcited by 105
  25. 2019GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verificationcited by 100
  26. 2021UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmarkcited by 98
  27. 2019ASER: A Large-scale Eventuality Knowledge Graphcited by 89
  28. 2021Can Generative Pre-trained Language Models Serve as Knowledge Bases for Closed-book QA?cited by 62
  29. 2021Broaden the Vision: Geo-Diverse Visual Commonsense Reasoningcited by 59
  30. 2020Temporal Reasoning on Implicit Events from Distant Supervisioncited by 58
  31. 2020TransOMCS: From Linguistic Graphs to Commonsense Knowledgecited by 52
  32. 2021Syntax-BERT: Improving Pre-trained Transformers with Syntax Treescited by 51
  33. 2020CSKG: The CommonSense Knowledge Graphcited by 45
  34. 2021UnitedQA: A Hybrid Approach for Open Domain Question Answeringcited by 42
  35. 2021Knowledge Enhanced Pretrained Language Models: A Compreshensive Surveycited by 33
  36. 2020SentiBERT: A Transferable Transformer-Based Architecture for Compositional Sentiment Semanticscited by 30
  37. 2021Men Are Elected, Women Are Married: Events Gender Bias on Wikipediacited by 30
  38. 2019How Additional Knowledge can Improve Natural Language Commonsense Question Answering?cited by 22

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 7; 2023: 13; 2024: 6; 2025: 3; 2026: 1.

30 papers.

Sort:

By area:

  1. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 663
  2. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  3. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  4. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  5. 2022K-LITE: Learning Transferable Visual Models with External Knowledgecited by 79
  6. 2022A Survey of Knowledge Enhanced Pre-trained Language Modelscited by 76
  7. 2023Generative Pre-trained Transformer: A Comprehensive Review on Enabling Technologies, Potential Applications, Emerging Challenges, and Future Directionscited by 55
  8. 2023Evaluating Large Language Models on Graphs: Performance Insights and Comparative Analysiscited by 35
  9. 2023How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advancescited by 33
  10. 2023The Life Cycle of Knowledge in Big Language Models: A Surveycited by 23
  11. 2023KILM: Knowledge Injection into Encoder-Decoder Language Modelscited by 19
  12. 2022ComFact: A Benchmark for Linking Contextual Commonsense Knowledgecited by 11
18 more
  1. 2023An Overview Of Temporal Commonsense Reasoning and Acquisitioncited by 10
  2. 2024Knowledge-aware Text-Image Retrieval for Remote Sensing Imagescited by 9
  3. 2023The Web Can Be Your Oyster for Improving Large Language Modelscited by 7
  4. 2023Plug-and-Play Knowledge Injection for Pre-trained Language Modelscited by 6
  5. 2024Tool Calling: Enhancing Medication Consultation via Retrieval-Augmented Large Language Modelscited by 6
  6. 2022A Survey of Text Representation Methods and Their Genealogycited by 5
  7. 2022Graph Learning and Its Advancements on Large Language Models: A Holistic Surveycited by 4
  8. 2024SeCG: Semantic-Enhanced 3D Visual Grounding via Cross-modal Graph Attentioncited by 4
  9. 2022A Survey on Knowledge-Enhanced Pre-trained Language Modelscited by 3
  10. 2023Thrust: Adaptively Propels Large Language Models with External Knowledgecited by 3
  11. 2025KnowPath: Knowledge-enhanced Reasoning via LLM-generated Inference Paths over Knowledge Graphscited by 3
  12. 2024Synergizing Machine Learning & Symbolic Methods: A Survey on Hybrid Approaches to Natural Language Processingcited by 1
  13. 2024Benchmarking Defeasible Reasoning with Large Language Models -- Initial Experiments and Future Directionscited by 1
  14. 2023KEPLET: Knowledge-Enhanced Pretrained Language Model with Topic Entity Awarenesscited by 0
  15. 2024LoGAH: Predicting 774-Million-Parameter Transformers using Graph HyperNetworks with 1/100 Parameterscited by 0
  16. 2025Towards Improving Interpretability of Language Model Generation through a Structured Knowledge Discovery Approachcited by 0
  17. 2025Knowledge Reasoning of Large Language Models Integrating Graph-Structured Information for Pest and Disease Control in Tobaccocited by 0
  18. 2026Graph-Augmented Reasoning with Large Language Models for Tobacco Pest and Disease Managementcited by 0