# A Unified Framework for Rank-based Evaluation Metrics for Link Prediction in Knowledge Graphs

arXiv: https://arxiv.org/abs/2203.07544 · HTML: https://mlsanity.com/paper/2203.07544/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 36 references that are in the map.
- First posted on arXiv 14 Mar 2022.

## Builds on (most cited first, 26 of 26)

- [Open Graph Benchmark: Datasets for Machine Learning on Graphs](https://mlsanity.com/paper/2005.00687/) (2020)
- [RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space](https://mlsanity.com/paper/1902.10197/) (2019)
- [Convolutional 2D Knowledge Graph Embeddings](https://mlsanity.com/paper/1707.01476/) (2017)
- [OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphs](https://mlsanity.com/paper/2103.09430/) (2021)
- [TuckER: Tensor Factorization for Knowledge Graph Completion](https://mlsanity.com/paper/1901.09590/) (2019)
- [Neural Bellman-Ford Networks: A General Graph Neural Network Framework for Link Prediction](https://mlsanity.com/paper/2106.06935/) (2021)
- [A Review of Relational Machine Learning for Knowledge Graphs](https://mlsanity.com/paper/1503.00759/) (2015)
- [Inductive Relation Prediction by Subgraph Reasoning](https://mlsanity.com/paper/1911.06962/) (2019)
- [Learning from positive and unlabeled data: a survey](https://mlsanity.com/paper/1811.04820/) (2018)
- [Query2box: Reasoning over Knowledge Graphs in Vector Space using Box Embeddings](https://mlsanity.com/paper/2002.05969/) (2020)
- [Beta Embeddings for Multi-Hop Logical Reasoning in Knowledge Graphs](https://mlsanity.com/paper/2010.11465/) (2020)
- [Embedding Logical Queries on Knowledge Graphs](https://mlsanity.com/paper/1806.01445/) (2018)
- [A Benchmarking Study of Embedding-based Entity Alignment for Knowledge Graphs](https://mlsanity.com/paper/2003.07743/) (2020)
- [Relation-Aware Entity Alignment for Heterogeneous Knowledge Graphs](https://mlsanity.com/paper/1908.08210/) (2019)
- [Complex Query Answering with Neural Link Predictors](https://mlsanity.com/paper/2011.03459/) (2020)
- [Deep Graph Matching Consensus](https://mlsanity.com/paper/2001.09621/) (2020)
- [Knowledge Graph Completion via Complex Tensor Factorization](https://mlsanity.com/paper/1702.06879/) (2017)
- [PyKEEN 1.0: A Python Library for Training and Evaluating Knowledge Graph Embeddings](https://mlsanity.com/paper/2007.14175/) (2020)
- [Bringing Light Into the Dark: A Large-scale Evaluation of Knowledge Graph Embedding Models Under a Unified Framework](https://mlsanity.com/paper/2006.13365/) (2020)
- [Relational Reflection Entity Alignment](https://mlsanity.com/paper/2008.07962/) (2020)
- [A Review of Biomedical Datasets Relating to Drug Discovery: A Knowledge Graph Perspective](https://mlsanity.com/paper/2102.10062/) (2021)
- [Answering Complex Queries in Knowledge Graphs with Bidirectional Sequence Encoders](https://mlsanity.com/paper/2004.02596/) (2020)
- [Query Embedding on Hyper-relational Knowledge Graphs](https://mlsanity.com/paper/2106.08166/) (2021)
- [On the Ambiguity of Rank-Based Evaluation of Entity Alignment or Link Prediction Methods](https://mlsanity.com/paper/2002.06914/) (2020)
- [On Evaluating Embedding Models for Knowledge Base Completion](https://mlsanity.com/paper/1810.07180/) (2018)
- [Improving Visual Relation Detection using Depth Maps](https://mlsanity.com/paper/1905.00966/) (2019)

## Built on by (most cited first, 15 of 15)

- [Long Range Graph Benchmark](https://mlsanity.com/paper/2206.08164/) (2022)
- [Sem@K: Is my knowledge graph embedding model semantic-aware?](https://mlsanity.com/paper/2301.05601/) (2023)
- [Predictive Multiplicity of Knowledge Graph Embeddings in Link Prediction](https://mlsanity.com/paper/2408.08226/) (2024)
- [On Large-scale Evaluation of Embedding Models for Knowledge Graph Completion](https://mlsanity.com/paper/2504.08970/) (2025)
- [Context-aware knowledge graph framework for traffic speed forecasting using graph neural network](https://mlsanity.com/paper/2407.17703/) (2024)
- [How Sharp and Bias-Robust is a Model? Dual Evaluation Perspectives on Knowledge Graph Completion](https://mlsanity.com/paper/2512.06296/) (2025)
- [QUEST: A robust attention formulation using query-modulated spherical attention](https://mlsanity.com/paper/2604.00199/) (2026)
- [Transcriptomics-based matching of drugs to diseases with deep learning](https://mlsanity.com/paper/2303.11695/) (2023)
- [Approximate Answering of Graph Queries](https://mlsanity.com/paper/2308.06585/) (2023)
- [Differential Encoding for Improved Representation Learning over Graphs](https://mlsanity.com/paper/2407.02758/) (2024)
- [Enabling Down Syndrome Research through a Knowledge Graph-Driven Analytical Framework](https://mlsanity.com/paper/2509.01565/) (2025)
- [HEHRGNN: A Unified Embedding Model for Knowledge Graphs with Hyperedges and Hyper-Relational Edges](https://mlsanity.com/paper/2602.18897/) (2026)
- [Generalized Rank-based Evaluation for Knowledge Graph Completion: Perspectives, Framework, and Analyses](https://mlsanity.com/paper/2606.08921/) (2026)
- [TRIAGE: Trustworthy Retrieval Instrumentation And Graph Evaluation](https://mlsanity.com/paper/2607.03447/) (2026)
- [Uncertainty in Representation Learning on Knowledge Graphs](https://mlsanity.com/paper/2610.06974/) (2026)

## Abstract (arXiv)

The link prediction task on knowledge graphs without explicit negative triples in the training data motivates the usage of rank-based metrics. Here, we review existing rank-based metrics and propose desiderata for improved metrics to address lack of interpretability and comparability of existing metrics to datasets of different sizes and properties. We introduce a simple theoretical framework for rank-based metrics upon which we investigate two avenues for improvements to existing metrics via alternative aggregation functions and concepts from probability theory. We finally propose several new rank-based metrics that are more easily interpreted and compared accompanied by a demonstration of their usage in a benchmarking of knowledge graph embedding models.