# When Metrics Disagree: A Meta-Analysis of Knowledge-Graph-Completion Model Benchmarking

arXiv: https://arxiv.org/abs/2606.10287 · HTML: https://mlsanity.com/paper/2606.10287/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 29 references that are in the map.
- First posted on arXiv 9 Jun 2026.

## Builds on (most cited first, 11 of 11)

- [RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space](https://mlsanity.com/paper/1902.10197/) (2019)
- [KG-BERT: BERT for Knowledge Graph Completion](https://mlsanity.com/paper/1909.03193/) (2019)
- [A Benchmarking Study of Embedding-based Entity Alignment for Knowledge Graphs](https://mlsanity.com/paper/2003.07743/) (2020)
- [A Re-evaluation of Knowledge Graph Completion Methods](https://mlsanity.com/paper/1911.03903/) (2019)
- [Knowledge Base Completion: Baselines Strike Back](https://mlsanity.com/paper/1705.10744/) (2017)
- [KERMIT: Knowledge Graph Completion of Enhanced Relation Modeling with Inverse Transformation](https://mlsanity.com/paper/2309.14770/) (2023)
- [A Relational Tucker Decomposition for Multi-Relational Link Prediction](https://mlsanity.com/paper/1902.00898/) (2019)
- [MuCo-KGC: Multi-Context-Aware Knowledge Graph Completion](https://mlsanity.com/paper/2503.03091/) (2025)
- [ReliK: A Reliability Measure for Knowledge Graph Embeddings](https://mlsanity.com/paper/2404.16572/) (2024)
- [KG-EDAS: A Meta-Metric Framework for Evaluating Knowledge Graph Completion Models](https://mlsanity.com/paper/2508.15357/) (2025)
- [How Sharp and Bias-Robust is a Model? Dual Evaluation Perspectives on Knowledge Graph Completion](https://mlsanity.com/paper/2512.06296/) (2025)

## Abstract (arXiv)

Evaluating Knowledge Graph Completion (KGC) models remains challenging because standard assessment relies on isolated rank-based metrics such as MRR, Hits@k, and Mean Rank, which often produce conflicting model orderings across datasets. A model that leads on MRR may trail on Hits@1, and strong performance on one dataset may not generalize to another. This fragmentation hinders comparison, enables selective reporting, and obscures real progress. We reframe KGC evaluation as a Multi-Criteria Decision-Making (MCDM) problem and present a meta-analysis of seven aggregators across five tests: consistency, cross-dataset stability, metric independence, robustness under noise, and generalizability. Each test is averaged over leave-one-model-out (LOMO) and leave-one-group-out (LOGO) removals so that reliability reflects aggregator behavior across diverse model subsets. Across tail (h,r,?) and relation (h,?,t) prediction, Pareto-optimal analysis identifies Z-score as the most balanced aggregator, which ranks DualE highest for tail prediction and FMS (Flow-Modulated Scoring) highest for relation prediction. A test-sensitivity analysis using the same removals shows that consistency and stability are largely removal-invariant, while generalizability and independence are the most sensitive. The framework resolves evaluation inconsistencies and offers evidence-based guidance for aggregator selection and model benchmarking in KGC.