mlsanityData to 14 Oct 2026

A Review on Language Models as Knowledge Bases

arXiv 2204.06031 ↗First posted 12 Apr 2022cs.CL cs.AIcited by 157

Not on the map. Why?

Abstract (arXiv)

Recently, there has been a surge of interest in the NLP community on the use of pretrained Language Models (LMs) as Knowledge Bases (KBs). Researchers have shown that LMs trained on a sufficiently large (web) corpus will encode a significant amount of knowledge implicitly in its parameters. The resulting LM can be probed for different kinds of knowledge and thus acting as a KB. This has a major advantage over traditional KBs in that this method requires no human supervision. In this paper, we present a set of aspects that we deem a LM should have to fully act as a KB, and review the recent literature with respect to those aspects.

Key facts. Cited by 157 papers in the map (data through 14 Oct 2026). Builds on 138 of its 182 references that are in the map. First posted on arXiv 12 Apr 2022.

Builds on

138 of 182 references.

References · 138

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  7. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  9. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  10. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  11. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  12. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
38 more
  1. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  2. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  3. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,367
  4. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  5. 2020Extracting Training Data from Large Language Modelscited by 2,189
  6. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  7. 2019Language Models as Knowledge Bases?cited by 1,936
  8. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  9. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  10. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  11. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  12. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  13. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  14. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  15. 2014Convolutional Neural Networks for Sentence Classificationcited by 1,127
  16. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  17. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  18. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  19. 2019How Can We Know What Language Models Know?cited by 959
  20. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 938
  21. 2019Attention is not Explanationcited by 929
  22. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 886
  23. 2019Are Sixteen Heads Really Better than One?cited by 816
  24. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  25. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 788
  26. 2016The Mythos of Model Interpretabilitycited by 721
  27. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 670
  28. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 669
  29. 2014Neural Turing Machinescited by 644
  30. 2020A Survey on Knowledge Graphs: Representation, Acquisition and Applicationscited by 643
  31. 2019Designing and Interpreting Probes with Control Taskscited by 640
  32. 2019Attention is not not Explanationcited by 636
  33. 2021Editing Factual Knowledge in Language Modelscited by 553
  34. 2019ERNIE: Enhanced Language Representation with Informative Entitiescited by 528
  35. 2021Fast Model Editing at Scalecited by 498
  36. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  37. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 438
  38. 2017Zero-Shot Relation Extraction via Reading Comprehensioncited by 426

50 most cited of 138 All 138 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 11; 2023: 54; 2024: 58; 2025: 28; 2026: 6.

157 papers.

Sort:

By area:

  1. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  2. 2022Mass-Editing Memory in a Transformercited by 1,001
  3. 2024HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Modelscited by 281
  4. 2024Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?cited by 232
  5. 2022Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Modelscited by 220
  6. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 206
  7. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 163
  8. 2023LLMs for Knowledge Graph Construction and Reasoning: Recent Capabilities and Future Opportunitiescited by 97
  9. 2023Can Knowledge Graphs Reduce Hallucinations in LLMs? : A Surveycited by 92
  10. 2023Crawling the Internal Knowledge-Base of Language Modelscited by 85
  11. 2023LoRAMoE: Alleviate World Knowledge Forgetting in Large Language Models via MoE-Style Plugincited by 82
  12. 2023Large Language Models and Knowledge Graphs: Opportunities and Challengescited by 71
38 more
  1. 2024How Large Language Models Encode Context Knowledge? A Layer-Wise Probing Studycited by 64
  2. 2024MoRAL: MoE Augmented LoRA for LLMs' Lifelong Learningcited by 63
  3. 2023GraphCare: Enhancing Healthcare Predictions with Personalized Knowledge Graphscited by 58
  4. 2023Give Us the Facts: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modelingcited by 56
  5. 2024UniBind: LLM-Augmented Unified and Balanced Representation Space to Bind Them Allcited by 49
  6. 2023SelfEvolve: A Code Evolution Framework via Large Language Modelscited by 43
  7. 2024On Large Language Models' Hallucination with Regard to Known Factscited by 43
  8. 2023Instruction-tuning Aligns LLMs to the Human Braincited by 40
  9. 2023Re-Reading Improves Reasoning in Large Language Modelscited by 37
  10. 2023SoK: Memorization in General-Purpose Large Language Modelscited by 37
  11. 2022DSI++: Updating Transformer Memory with New Documentscited by 35
  12. 2023How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advancescited by 33
  13. 2023Give Me the Facts! A Survey on Factual Knowledge Probing in Pre-trained Language Modelscited by 32
  14. 2024Logically Consistent Language Models via Neuro-Symbolic Integrationcited by 30
  15. 2025From Language to Cognition: How LLMs Outgrow the Human Language Networkcited by 30
  16. 2024Undesirable Memorization in Large Language Models: A Surveycited by 29
  17. 2025ECLeKTic: a Novel Challenge Set for Evaluation of Cross-Lingual Knowledge Transfercited by 29
  18. 2023Emptying the Ocean with a Spoon: Should We Edit Models?cited by 27
  19. 2023Language Model Analysis for Ontology Subsumption Inferencecited by 25
  20. 2023A Glitch in the Matrix? Locating and Detecting Language Model Grounding with Fakepediacited by 24
  21. 2024Fundamental Problems With Model Editing: How Should Rational Belief Revision Work in LLMs?cited by 24
  22. 2022FolkScope: Intention Knowledge Graph Construction for E-commerce Commonsense Discoverycited by 23
  23. 2023The Life Cycle of Knowledge in Big Language Models: A Surveycited by 23
  24. 2023Dynamic Benchmarking of Masked Language Models on Temporal Concept Drift with Multiple Viewscited by 21
  25. 2022BertNet: Harvesting Knowledge Graphs with Arbitrary Relations from Pretrained Language Modelscited by 20
  26. 2024Why Does New Knowledge Create Messy Ripple Effects in LLMs?cited by 18
  27. 2024Can Language Models Act as Knowledge Bases at Scale?cited by 17
  28. 2023Measuring and Modifying Factual Knowledge in Large Language Modelscited by 16
  29. 2023Discovering Knowledge-Critical Subnetworks in Pretrained Language Modelscited by 16
  30. 2025Knowledge Updating? No More Model Editing! Just Selective Contextual Reasoningcited by 16
  31. 2025Classroom Simulacra: Building Contextual Student Generative Agents in Online Education for Learning Behavioral Simulationcited by 15
  32. 2025Extracting Probabilistic Knowledge from Large Language Models for Bayesian Network Parameterizationcited by 15
  33. 2023Understanding Finetuning for Factual Knowledge Extraction from Language Modelscited by 13
  34. 2023Preserving Commonsense Knowledge from Pre-trained Language Models via Causal Inferencecited by 13
  35. 2023Do PLMs Know and Understand Ontological Knowledge?cited by 13
  36. 2023In-Context Exemplars as Clues to Retrieving from Large Associative Memorycited by 13
  37. 2025Learning Facts at Scale with Active Readingcited by 13
  38. 2023Localized Symbolic Knowledge Distillation for Visual Commonsense Modelscited by 12

50 most cited of 157 All 157 (JSON)