# Realistic Re-evaluation of Knowledge Graph Completion Methods: An Experimental Study

arXiv: https://arxiv.org/abs/2003.08001 · HTML: https://mlsanity.com/paper/2003.08001/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Mar 2020.

## Built on by (most cited first, 30 of 37)

- [Knowledge Graphs: Opportunities and Challenges](https://mlsanity.com/paper/2303.13948/) (2023)
- [Dipping PLMs Sauce: Bridging Structure and Text for Effective Knowledge Graph Completion via Conditional Soft Prompting](https://mlsanity.com/paper/2307.01709/) (2023)
- [Knowledge Is Flat: A Seq2Seq Generative Framework for Various Knowledge Graph Completion](https://mlsanity.com/paper/2209.07299/) (2022)
- [Contextualization Distillation from Large Language Model for Knowledge Graph Completion](https://mlsanity.com/paper/2402.01729/) (2024)
- [Rethinking Knowledge Graph Evaluation Under the Open-World Assumption](https://mlsanity.com/paper/2209.08858/) (2022)
- [Efficient Non-Sampling Knowledge Graph Embedding](https://mlsanity.com/paper/2104.10796/) (2021)
- [ExpressivE: A Spatio-Functional Embedding For Knowledge Graph Completion](https://mlsanity.com/paper/2206.04192/) (2022)
- [Automated Benchmark Auditing for AI Agents and Large Language Models](https://mlsanity.com/paper/2605.26079/) (2026)
- [Neural Graph Databases](https://mlsanity.com/paper/2209.09732/) (2022)
- [Are Missing Links Predictable? An Inferential Benchmark for Knowledge Graph Completion](https://mlsanity.com/paper/2108.01387/) (2021)
- [Multiple Run Ensemble Learning with Low-Dimensional Knowledge Graph Embeddings](https://mlsanity.com/paper/2104.05003/) (2021)
- [Re-thinking Knowledge Graph Completion Evaluation from an Information Retrieval Perspective](https://mlsanity.com/paper/2205.04105/) (2022)
- [Deep Active Alignment of Knowledge Graph Entities and Schemata](https://mlsanity.com/paper/2304.04389/) (2023)
- [Knowledge Base Completion for Long-Tail Entities](https://mlsanity.com/paper/2306.17472/) (2023)
- [KGrEaT: A Framework to Evaluate Knowledge Graphs via Downstream Tasks](https://mlsanity.com/paper/2308.10537/) (2023)
- [Joint Pre-training and Local Re-training: Transferable Representation Learning on Multi-source Knowledge Graphs](https://mlsanity.com/paper/2306.02679/) (2023)
- [MoCoSA: Momentum Contrast for Knowledge Graph Completion with Structure-Augmented Pre-trained Language Models](https://mlsanity.com/paper/2308.08204/) (2023)
- [CausalLP: Learning causal relations with weighted knowledge graph link prediction](https://mlsanity.com/paper/2405.02327/) (2024)
- [KGExplainer: Towards Exploring Connected Subgraph Explanations for Knowledge Graph Completion](https://mlsanity.com/paper/2404.03893/) (2024)
- [What is Learned in Knowledge Graph Embeddings?](https://mlsanity.com/paper/2110.09978/) (2021)
- [Revisiting Inferential Benchmarks for Knowledge Graph Completion](https://mlsanity.com/paper/2306.04814/) (2023)
- [Event Prediction using Case-Based Reasoning over Knowledge Graphs](https://mlsanity.com/paper/2309.12423/) (2023)
- [Trust, Accountability, and Autonomy in Knowledge Graph-based AI for Self-determination](https://mlsanity.com/paper/2310.19503/) (2023)
- [On Large-scale Evaluation of Embedding Models for Knowledge Graph Completion](https://mlsanity.com/paper/2504.08970/) (2025)
- [In-depth analysis of music structure as a text network](https://mlsanity.com/paper/2303.13631/) (2023)
- [Towards Better Benchmark Datasets for Inductive Knowledge Graph Completion](https://mlsanity.com/paper/2406.11898/) (2024)
- [Two-dimensional Taxonomy for N-ary Knowledge Representation Learning Methods](https://mlsanity.com/paper/2506.05626/) (2025)
- [KG-EDAS: A Meta-Metric Framework for Evaluating Knowledge Graph Completion Models](https://mlsanity.com/paper/2508.15357/) (2025)
- [Effects of Locality and Rule Language on Explanations for Knowledge Graph Embeddings](https://mlsanity.com/paper/2302.06967/) (2023)
- [Prior Bilinear Based Models for Knowledge Graph Completion](https://mlsanity.com/paper/2309.13834/) (2023)

## Abstract (arXiv)

In the active research area of employing embedding models for knowledge graph completion, particularly for the task of link prediction, most prior studies used two benchmark datasets FB15k and WN18 in evaluating such models. Most triples in these and other datasets in such studies belong to reverse and duplicate relations which exhibit high data redundancy due to semantic duplication, correlation or data incompleteness. This is a case of excessive data leakage---a model is trained using features that otherwise would not be available when the model needs to be applied for real prediction. There are also Cartesian product relations for which every triple formed by the Cartesian product of applicable subjects and objects is a true fact. Link prediction on the aforementioned relations is easy and can be achieved with even better accuracy using straightforward rules instead of sophisticated embedding models. A more fundamental defect of these models is that the link prediction scenario, given such data, is non-existent in the real-world. This paper is the first systematic study with the main objective of assessing the true effectiveness of embedding models when the unrealistic triples are removed. Our experiment results show these models are much less accurate than what we used to perceive. Their poor accuracy renders link prediction a task without truly effective automated solution. Hence, we call for re-investigation of possible effective approaches.