mlsanityData to 14 Oct 2026

Evaluating the Knowledge Base Completion Potential of GPT

arXiv 2310.14771 ↗First posted 23 Oct 2023cs.CL cs.AIcited by 9

Not on the map. Why?

Abstract (arXiv)

Structured knowledge bases (KBs) are an asset for search engines and other applications, but are inevitably incomplete. Language models (LMs) have been proposed for unsupervised knowledge base completion (KBC), yet, their ability to do this at scale and with high accuracy remains an open question. Prior experimental studies mostly fall short because they only evaluate on popular subjects, or sample already existing facts from KBs. In this work, we perform a careful evaluation of GPT's potential to complete the largest public KB: Wikidata. We find that, despite their size and capabilities, models like GPT-3, ChatGPT and GPT-4 do not achieve fully convincing results on this task. Nonetheless, they provide solid improvements over earlier approaches with smaller LMs. In particular, we show that, with proper thresholding, GPT-3 enables to extend Wikidata by 27M facts at 90% precision.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 17 of its 29 references that are in the map. First posted on arXiv 23 Oct 2023.

Builds on

17 of 29 references.

References · 17

  1. 2019Language Models as Knowledge Bases?cited by 1,936
  2. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,128
  3. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  4. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 719
  5. 2021Editing Factual Knowledge in Language Modelscited by 553
  6. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 445
  7. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 364
  8. 2020How Context Affects Language Models' Factual Predictionscited by 181
  9. 2023Head-to-Tail: How Knowledgeable are Large Language Models (LLMs)? A.K.A. Will LLMs Replace Knowledge Graphs?cited by 168
  10. 2020Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queriescited by 107
  11. 2020CoDEx: A Comprehensive Knowledge Graph Completion Benchmarkcited by 102
  12. 2019E-BERT: Efficient-Yet-Effective Entity Embeddings for BERTcited by 96
5 more
  1. 2023Crawling the Internal Knowledge-Base of Language Modelscited by 85
  2. 2020Machine Knowledge: Creation and Curation of Comprehensive Knowledge Basescited by 55
  3. 2022Prompting as Probing: Using Language Models for Knowledge Base Constructioncited by 32
  4. 2023Evaluating Language Models for Knowledge Base Completioncited by 20
  5. 2023Extracting Multi-valued Relations from Language Modelscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 3; 2025: 4; 2026: 1.

9 papers.

  1. 2023Can Knowledge Graphs Reduce Hallucinations in LLMs? : A Surveycited by 92
  2. 2024KAG: Boosting LLMs in Professional Domains via Knowledge Augmented Generationcited by 67
  3. 2024Can Language Models Act as Knowledge Bases at Scale?cited by 17
  4. 2025Knowledge Base Construction for Knowledge-Augmented Text-to-SQLcited by 9
  5. 2025Large Language Models, Knowledge Graphs and Search Engines: A Crossroads for Answering Users' Questionscited by 8
  6. 2025GPTKB v1.5: A Massive Knowledge Base for Exploring Factual LLM Knowledgecited by 5
  7. 2025Auditing a KB Elicitation of Frontier LLM Knowledge: A Multi-dimensional Analysis of GPTKB v1.5cited by 3
  8. 2024Recall Them All: Retrieval-Augmented Language Models for Long Object List Extraction from Long Documentscited by 1
  9. 2026Toward a Gricean Retreat: Probing LLMs for Knowledge Boundaries and Referent Specificitycited by 0