# Evaluating Language Models for Knowledge Base Completion

arXiv: https://arxiv.org/abs/2303.11082 · HTML: https://mlsanity.com/paper/2303.11082/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 28 references that are in the map.
- First posted on arXiv 20 Mar 2023.

## Builds on (most cited first, 14 of 14)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Convolutional 2D Knowledge Graph Embeddings](https://mlsanity.com/paper/1707.01476/) (2017)
- [How Much Knowledge Can You Pack Into the Parameters of a Language Model?](https://mlsanity.com/paper/2002.08910/) (2020)
- [Editing Factual Knowledge in Language Models](https://mlsanity.com/paper/2104.08164/) (2021)
- [Measuring and Improving Consistency in Pretrained Language Models](https://mlsanity.com/paper/2102.01017/) (2021)
- [How Context Affects Language Models' Factual Predictions](https://mlsanity.com/paper/2005.04611/) (2020)
- [Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queries](https://mlsanity.com/paper/2008.09036/) (2020)
- [CoDEx: A Comprehensive Knowledge Graph Completion Benchmark](https://mlsanity.com/paper/2009.07810/) (2020)
- [E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT](https://mlsanity.com/paper/1911.03681/) (2019)
- [Crawling the Internal Knowledge-Base of Language Models](https://mlsanity.com/paper/2301.12810/) (2023)
- [Machine Knowledge: Creation and Curation of Comprehensive Knowledge Bases](https://mlsanity.com/paper/2009.11564/) (2020)
- [Analyzing Race and Country of Citizenship Bias in Wikidata](https://mlsanity.com/paper/2108.05412/) (2021)

## Built on by (most cited first, 20 of 20)

- [Large Language Models and Knowledge Graphs: Opportunities and Challenges](https://mlsanity.com/paper/2308.06374/) (2023)
- [Text2KGBench: A Benchmark for Ontology-Driven Knowledge Graph Generation from Text](https://mlsanity.com/paper/2308.02357/) (2023)
- [From human experts to machines: An LLM supported approach to ontology and knowledge graph construction](https://mlsanity.com/paper/2403.08345/) (2024)
- [NetSafe: Exploring the Topological Safety of Multi-agent Networks](https://mlsanity.com/paper/2410.15686/) (2024)
- [Towards Ontology Construction with Language Models](https://mlsanity.com/paper/2309.09898/) (2023)
- [Can Language Models Act as Knowledge Bases at Scale?](https://mlsanity.com/paper/2402.14273/) (2024)
- [Rethinking Language Models as Symbolic Knowledge Graphs](https://mlsanity.com/paper/2308.13676/) (2023)
- [Enabling LLM Knowledge Analysis via Extensive Materialization](https://mlsanity.com/paper/2411.04920/) (2024)
- [Knowledge Graphs for the Life Sciences: Recent Developments, Challenges and Opportunities](https://mlsanity.com/paper/2309.17255/) (2023)
- [Evaluating the Knowledge Base Completion Potential of GPT](https://mlsanity.com/paper/2310.14771/) (2023)
- [Positional Biases Shift as Inputs Approach Context Window Limits](https://mlsanity.com/paper/2508.07479/) (2025)
- [Completeness, Recall, and Negation in Open-World Knowledge Bases: A Survey](https://mlsanity.com/paper/2305.05403/) (2023)
- [Using Large Language Models for Knowledge Engineering (LLMKE): A Case Study on Wikidata](https://mlsanity.com/paper/2309.08491/) (2023)
- [A Language Model based Framework for New Concept Placement in Ontologies](https://mlsanity.com/paper/2402.17897/) (2024)
- [InFact: Informativeness Alignment for Improved LLM Factuality](https://mlsanity.com/paper/2505.20487/) (2025)
- [Large Language Model-Enhanced Symbolic Reasoning for Knowledge Base Completion](https://mlsanity.com/paper/2501.01246/) (2025)
- [Multimodal Cultural Heritage Knowledge Graph Extension with Language and Vision Models](https://mlsanity.com/paper/2605.17669/) (2026)
- [What Current AI Benchmarks Leave Unmeasured: Modality, Search, Citations, and Implications (for Safety Evaluations)](https://mlsanity.com/paper/2608.06202/) (2026)
- [Noise or Nuance: An Investigation Into Useful Information and Filtering For LLM Driven AKBC](https://mlsanity.com/paper/2509.08903/) (2025)
- [SWORD: Wikidata-based Distortions Reveal Hidden Cross-Lingual Inconsistencies in LLM Factual Error Rejection](https://mlsanity.com/paper/2609.09349/) (2026)

## Abstract (arXiv)

Structured knowledge bases (KBs) are a foundation of many intelligent applications, yet are notoriously incomplete. Language models (LMs) have recently been proposed for unsupervised knowledge base completion (KBC), yet, despite encouraging initial results, questions regarding their suitability remain open. Existing evaluations often fall short because they only evaluate on popular subjects, or sample already existing facts from KBs. In this work, we introduce a novel, more challenging benchmark dataset, and a methodology tailored for a realistic assessment of the KBC potential of LMs. For automated assessment, we curate a dataset called WD-KNOWN, which provides an unbiased random sample of Wikidata, containing over 3.9 million facts. In a second step, we perform a human evaluation on predictions that are not yet in the KB, as only this provides real insights into the added value over existing KBs. Our key finding is that biases in dataset conception of previous benchmarks lead to a systematic overestimate of LM performance for KBC. However, our results also reveal strong areas of LMs. We could, for example, perform a significant completion of Wikidata on the relations nativeLanguage, by a factor of ~21 (from 260k to 5.8M) at 82% precision, usedLanguage, by a factor of ~2.1 (from 2.1M to 6.6M) at 82% precision, and citizenOf by a factor of ~0.3 (from 4.2M to 5.3M) at 90% precision. Moreover, we find that LMs possess surprisingly strong generalization capabilities: even on relations where most facts were not directly observed in LM training, prediction quality can be high.