# Generalized Rank-based Evaluation for Knowledge Graph Completion: Perspectives, Framework, and Analyses

arXiv: https://arxiv.org/abs/2606.08921 · HTML: https://mlsanity.com/paper/2606.08921/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Knowledge-graph and structured-knowledge RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 35 of its 64 references that are in the map.
- First posted on arXiv 8 Jun 2026.

## Placed in

- Retrieval-augmented generation / [Knowledge-graph and structured-knowledge RAG](https://mlsanity.com/rag/knowledge-graph-and-structured-knowledge-rag/)

## Builds on (most cited first, 30 of 35)

- [Modeling Relational Data with Graph Convolutional Networks](https://mlsanity.com/paper/1703.06103/) (2017)
- [From Local to Global: A Graph RAG Approach to Query-Focused Summarization](https://mlsanity.com/paper/2404.16130/) (2024)
- [Embedding Entities and Relations for Learning and Inference in Knowledge Bases](https://mlsanity.com/paper/1412.6575/) (2014)
- [Complex Embeddings for Simple Link Prediction](https://mlsanity.com/paper/1606.06357/) (2016)
- [RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space](https://mlsanity.com/paper/1902.10197/) (2019)
- [Convolutional 2D Knowledge Graph Embeddings](https://mlsanity.com/paper/1707.01476/) (2017)
- [Unifying Large Language Models and Knowledge Graphs: A Roadmap](https://mlsanity.com/paper/2306.08302/) (2023)
- [Composition-based Multi-Relational Graph Convolutional Networks](https://mlsanity.com/paper/1911.03082/) (2019)
- [QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering](https://mlsanity.com/paper/2104.06378/) (2021)
- [TuckER: Tensor Factorization for Knowledge Graph Completion](https://mlsanity.com/paper/1901.09590/) (2019)
- [Neural Bellman-Ford Networks: A General Graph Neural Network Framework for Link Prediction](https://mlsanity.com/paper/2106.06935/) (2021)
- [Variational Reasoning for Question Answering with Knowledge Graph](https://mlsanity.com/paper/1709.04071/) (2017)
- [DKN: Deep Knowledge-Aware Network for News Recommendation](https://mlsanity.com/paper/1801.08284/) (2018)
- [Knowledge Graph Convolutional Networks for Recommender Systems](https://mlsanity.com/paper/1904.12575/) (2019)
- [RNNLogic: Learning Logic Rules for Reasoning on Knowledge Graphs](https://mlsanity.com/paper/2010.04029/) (2020)
- [CoDEx: A Comprehensive Knowledge Graph Completion Benchmark](https://mlsanity.com/paper/2009.07810/) (2020)
- [Knowledge Graph Reasoning with Relational Digraph](https://mlsanity.com/paper/2108.06040/) (2021)
- [One-Shot Relational Learning for Knowledge Graphs](https://mlsanity.com/paper/1808.09040/) (2018)
- [A Re-evaluation of Knowledge Graph Completion Methods](https://mlsanity.com/paper/1911.03903/) (2019)
- [Probabilistic Logic Neural Networks for Reasoning](https://mlsanity.com/paper/1906.08495/) (2019)
- [HyperGraphRAG: Retrieval-Augmented Generation via Hypergraph-Structured Knowledge Representation](https://mlsanity.com/paper/2503.21322/) (2025)
- [PathRAG: Pruning Graph-based Retrieval Augmented Generation with Relational Paths](https://mlsanity.com/paper/2502.14902/) (2025)
- [Interactive Recommender System via Knowledge Graph-enhanced Reinforcement Learning](https://mlsanity.com/paper/2006.10389/) (2020)
- [A Review of Biomedical Datasets Relating to Drug Discovery: A Knowledge Graph Perspective](https://mlsanity.com/paper/2102.10062/) (2021)
- [Drug Repurposing for COVID-19 via Knowledge Graph Completion](https://mlsanity.com/paper/2010.09600/) (2020)
- [HousE: Knowledge Graph Embedding with Householder Parameterization](https://mlsanity.com/paper/2202.07919/) (2022)
- [KCD: Knowledge Walks and Textual Cues Enhanced Political Perspective Detection in News Media](https://mlsanity.com/paper/2204.04046/) (2022)
- [Tackling Long-Tailed Relations and Uncommon Entities in Knowledge Graph Completion](https://mlsanity.com/paper/1909.11359/) (2019)
- [Rethinking Knowledge Graph Evaluation Under the Open-World Assumption](https://mlsanity.com/paper/2209.08858/) (2022)
- [A Unified Framework for Rank-based Evaluation Metrics for Link Prediction in Knowledge Graphs](https://mlsanity.com/paper/2203.07544/) (2022)

## Abstract (arXiv)

Knowledge graph completion (KGC) aims to predict missing facts from an observed knowledge graph (KG), playing a crucial role in a wide range of real-world applications such as drug discovery, recommender systems, and retrieval-augmented generation (RAG). Although numerous KGC models have been proposed, the evaluation of KGC remains underexplored, despite its critical role in reliably assessing model performance and selecting appropriate models for real-world applications. In this paper, we introduce two important perspectives for KGC evaluation that are overlooked by existing evaluation metrics, (P1) predictive sharpness and (P2) popularity-bias robustness. To address both perspectives, we propose a generalized evaluation framework, PROBE, which consists of a rank transformer (RT) that estimates the score of each prediction based on a desired level of predictive sharpness and a rank aggregator (RA) that determines the final evaluation score by aggregating all prediction scores according to a desired level of popularity-bias robustness. We theoretically analyze PROBE by defining six key properties for reliable KGC evaluation and prove that PROBE satisfies all the properties, while existing metrics fail to satisfy some. In particular, due to the open-world nature of KGs, an evaluation metric should preserve the relative performance of KGC models even when only incomplete facts are observed. We show that PROBE better maintains such consistency, providing a more reliable estimate of intrinsic model performance than existing metrics. Extensive experiments with six KGC models on six real-world KGs reveal that existing metrics may over- or under-estimate model performance depending on different evaluation perspectives, whereas PROBE enables a more comprehensive, flexible, and consistent evaluation of KGC models.