mlsanityData to 7 Oct 2026

Language Models as Knowledge Bases?

arXiv 1909.01066 ↗First posted 3 Sep 2019cs.CLcited by 1,922

Not on the map. Why?

Cited by 102 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent progress in pretraining language models on large textual corpora led to a surge of improvements for downstream NLP tasks. Whilst learning linguistic knowledge, these models may also be storing relational knowledge present in the training data, and may be able to answer queries structured as "fill-in-the-blank" cloze statements. Language models have many advantages over structured knowledge bases: they require no schema engineering, allow practitioners to query about an open class of relations, are easy to extend to more data, and require no human supervision to train. We present an in-depth analysis of the relational knowledge already present (without fine-tuning) in a wide range of state-of-the-art pretrained language models. We find that (i) without fine-tuning, BERT contains relational knowledge competitive with traditional NLP methods that have some access to oracle knowledge, (ii) BERT also does remarkably well on open-domain question answering against a supervised baseline, and (iii) certain types of factual knowledge are learned much more readily than others by standard language model pretraining approaches. The surprisingly strong ability of these models to recall factual knowledge without any fine-tuning demonstrates their potential as unsupervised open-domain QA systems. The code to reproduce our analysis is available at https://github.com/facebookresearch/LAMA.

Key facts. Cited by 1,922 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 Sep 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 2; 2021: 253; 2022: 344; 2023: 468; 2024: 436; 2025: 243; 2026: 176.

1,922 papers.

Sort:

By area:

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  2. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  4. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  5. 2023A Survey of Large Language Modelscited by 2,537
  6. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  7. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  8. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  9. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  10. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  11. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  12. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
38 more
  1. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  2. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  3. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  4. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  5. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  6. 2022Mass-Editing Memory in a Transformercited by 976
  7. 2023Steering Language Models With Activation Engineeringcited by 824
  8. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 814
  9. 2022Galactica: A Large Language Model for Sciencecited by 736
  10. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  11. 2021GPT Understands, Toocited by 715
  12. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
  13. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 663
  14. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  15. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  16. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  17. 2021Editing Factual Knowledge in Language Modelscited by 548
  18. 2024RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrievalcited by 519
  19. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  20. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  21. 2023Active Retrieval Augmented Generationcited by 506
  22. 2022Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Modelscited by 501
  23. 2021Fast Model Editing at Scalecited by 491
  24. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  25. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
  26. 2021MetaICL: Learning to Learn In Contextcited by 450
  27. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 450
  28. 2022Knowledge Unlearning for Mitigating Privacy Risks in Language Modelscited by 424
  29. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
  30. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 399
  31. 2021QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answeringcited by 386
  32. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 376
  33. 2021A Survey on Automated Fact-Checkingcited by 376
  34. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 361
  35. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
  36. 2021Entity-Based Knowledge Conflicts in Question Answeringcited by 349
  37. 2021True Few-Shot Learning with Language Modelscited by 343
  38. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 333

50 most cited of 1,922 All 1,922 (JSON)