mlsanityData to 14 Oct 2026

Crawling the Internal Knowledge-Base of Language Models

arXiv 2301.12810 ↗First posted 30 Jan 2023cs.CL cs.AIcited by 85

Not on the map. Why?

Abstract (arXiv)

Language models are trained on large volumes of text, and as a result their parameters might contain a significant body of factual knowledge. Any downstream task performed by these models implicitly builds on these facts, and thus it is highly desirable to have means for representing this body of knowledge in an interpretable way. However, there is currently no mechanism for such a representation. Here, we propose to address this goal by extracting a knowledge-graph of facts from a given language model. We describe a procedure for ``crawling'' the internal knowledge-base of a language model. Specifically, given a seed entity, we expand a knowledge-graph around it. The crawling procedure is decomposed into sub-tasks, realized through specially designed prompts that control for both precision (i.e., that no wrong facts are generated) and recall (i.e., the number of facts generated). We evaluate our approach on graphs crawled starting from dozens of seed entities, and show it yields high precision graphs (82-92%), while emitting a reasonable number of facts per entity.

Key facts. Cited by 85 papers in the map (data through 14 Oct 2026). Builds on 18 of its 27 references that are in the map. First posted on arXiv 30 Jan 2023.

Builds on

18 of 27 references.

References · 18

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  5. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  6. 2019Language Models as Knowledge Bases?cited by 1,936
  7. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  8. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,067
  9. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
  10. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 719
  11. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 558
  12. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 445
6 more
  1. 2020Calibration of Pre-trained Transformerscited by 315
  2. 2022Transformer Memory as a Differentiable Search Indexcited by 247
  3. 2022A Review on Language Models as Knowledge Basescited by 157
  4. 2022Compositional Semantic Parsing with Large Language Modelscited by 95
  5. 2022Investigating Selective Prediction Approaches Across Several Tasks in IID, OOD, and Adversarial Settingscited by 66
  6. 2022Prompting as Probing: Using Language Models for Knowledge Base Constructioncited by 32

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 30; 2024: 26; 2025: 20; 2026: 9.

85 papers.

Sort:

By area:

  1. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 528
  2. 2023Evaluating the Ripple Effects of Knowledge Editing in Language Modelscited by 242
  3. 2024Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?cited by 232
  4. 2023Linearity of Relation Decoding in Transformer Language Modelscited by 200
  5. 2024Don't Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaborationcited by 180
  6. 2023Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMscited by 178
  7. 2023LM vs LM: Detecting Factual Errors via Cross Examinationcited by 156
  8. 2023Self-contradictory Hallucinations of Large Language Models: Evaluation, Detection and Mitigationcited by 124
  9. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 118
  10. 2025A Survey of Graph Retrieval-Augmented Generation for Customized Large Language Modelscited by 88
  11. 2023Learning from models beyond fine-tuningcited by 48
  12. 2024On Large Language Models' Hallucination with Regard to Known Factscited by 43
38 more
  1. 2023The Unequal Opportunities of Large Language Models: Revealing Demographic Bias through Job Recommendationscited by 41
  2. 2024From human experts to machines: An LLM supported approach to ontology and knowledge graph constructioncited by 40
  3. 2024Estimating Knowledge in Large Language Models Without Generating a Single Tokencited by 40
  4. 2024Improving Uncertainty Estimation through Semantically Diverse Language Generationcited by 35
  5. 2024SAC-KG: Exploiting Large Language Models as Skilled Automatic Constructors for Domain Knowledge Graphscited by 31
  6. 2024I Don't Know: Explicit Modeling of Uncertainty with an [IDK] Tokencited by 26
  7. 2024DyKnow: Dynamically Verifying Time-Sensitive Factual Knowledge in LLMscited by 24
  8. 2023Towards Ontology Construction with Language Modelscited by 23
  9. 2023Evaluating Language Models for Knowledge Base Completioncited by 20
  10. 2024Rethinking Uncertainty Estimation in LLMs: A Principled Single-Sequence Measurecited by 18
  11. 2024Self-Tuning: Instructing LLMs to Effectively Acquire New Knowledge through Self-Teachingcited by 17
  12. 2023Measuring and Modifying Factual Knowledge in Large Language Modelscited by 16
  13. 2023Kiki or Bouba? Sound Symbolism in Vision-and-Language Modelscited by 14
  14. 2023Evaluating Dependencies in Fact Editing for Language Models: Specificity and Implication Awarenesscited by 14
  15. 2023Rethinking Language Models as Symbolic Knowledge Graphscited by 12
  16. 2024Enabling LLM Knowledge Analysis via Extensive Materializationcited by 12
  17. 2025Knowledge-Instruct: Effective Continual Pre-training from Limited Data using Instructionscited by 12
  18. 2023Large Language Model Programscited by 11
  19. 2023Selenite: Scaffolding Online Sensemaking with Comprehensive Overviews Elicited from Large Language Modelscited by 10
  20. 2025LLMs as Repositories of Factual Knowledge: Limitations and Solutionscited by 10
  21. 2023Beyond Testers' Biases: Guiding Model Testing with Knowledge Bases using LLMscited by 9
  22. 2023Evaluating the Knowledge Base Completion Potential of GPTcited by 9
  23. 2024Continual Memorization of Factoids in Language Modelscited by 8
  24. 2024When to Speak, When to Abstain: Contrastive Decoding with Abstentioncited by 8
  25. 2025How new data permeates LLM knowledge and how to dilute itcited by 8
  26. 2023Has It All Been Solved? Open NLP Research Questions Not Solved by Large Language Modelscited by 7
  27. 2024Desiderata for the Context Use of Question Answering Systemscited by 7
  28. 2025Will Pre-Training Ever End? A First Step Toward Next-Generation Foundation MLLMs via Self-Improving Systematic Cognitioncited by 7
  29. 2023"Im not Racist but...": Discovering Bias in the Internal Knowledge of Large Language Modelscited by 5
  30. 2023Making Large Language Models Better Knowledge Miners for Online Marketing with Progressive Prompting Augmentationcited by 5
  31. 2025GPTKB v1.5: A Massive Knowledge Base for Exploring Factual LLM Knowledgecited by 5
  32. 2023KGQuiz: Evaluating the Generalization of Encoded Knowledge in Large Language Modelscited by 4
  33. 2023Defining a New NLP Playgroundcited by 4
  34. 2024Meta-Cognitive Analysis: Evaluating Declarative and Procedural Knowledge in Datasets and Large Language Modelscited by 4
  35. 2024Learning and Unlearning of Fabricated Knowledge in Language Modelscited by 4
  36. 2025Demystifying Scientific Problem-Solving in LLMs by Probing Knowledge and Reasoningcited by 4
  37. 2023Extracting Multi-valued Relations from Language Modelscited by 3
  38. 2024LIDAO: Towards Limited Interventions for Debiasing (Large) Language Modelscited by 3

50 most cited of 85 All 85 (JSON)