# Bringing Light Into the Dark: A Large-scale Evaluation of Knowledge Graph Embedding Models Under a Unified Framework

arXiv: https://arxiv.org/abs/2006.13365 · HTML: https://mlsanity.com/paper/2006.13365/

## Key facts

- Cited by 70 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Jun 2020.

## Built on by (most cited first, 30 of 70)

- [A Survey of Knowledge Graph Reasoning on Graph Types: Static, Dynamic, and Multimodal](https://mlsanity.com/paper/2212.05767/) (2022)
- [Towards Foundation Models for Knowledge Graph Reasoning](https://mlsanity.com/paper/2310.04562/) (2023)
- [NodePiece: Compositional and Parameter-Efficient Representations of Large Knowledge Graphs](https://mlsanity.com/paper/2106.12144/) (2021)
- [Evaluating Graph Neural Networks for Link Prediction: Current Pitfalls and New Benchmarking](https://mlsanity.com/paper/2306.10453/) (2023)
- [A Decade of Knowledge Graphs in Natural Language Processing: A Survey](https://mlsanity.com/paper/2210.00105/) (2022)
- [A Review of Biomedical Datasets Relating to Drug Discovery: A Knowledge Graph Perspective](https://mlsanity.com/paper/2102.10062/) (2021)
- [Neural Graph Reasoning: Complex Logical Query Answering Meets Graph Databases](https://mlsanity.com/paper/2303.14617/) (2023)
- [Weisfeiler and Leman Go Relational](https://mlsanity.com/paper/2211.17113/) (2022)
- [An Open Challenge for Inductive Link Prediction on Knowledge Graphs](https://mlsanity.com/paper/2203.01520/) (2022)
- [Inductive Logical Query Answering in Knowledge Graphs](https://mlsanity.com/paper/2210.08008/) (2022)
- [Query Embedding on Hyper-relational Knowledge Graphs](https://mlsanity.com/paper/2106.08166/) (2021)
- [Improving Inductive Link Prediction Using Hyper-Relational Facts](https://mlsanity.com/paper/2107.04894/) (2021)
- [Visual Question Answering: A Survey on Techniques and Common Trends in Recent Literature](https://mlsanity.com/paper/2305.11033/) (2023)
- [Understanding the Performance of Knowledge Graph Embeddings in Drug Discovery](https://mlsanity.com/paper/2105.10488/) (2021)
- [A Unified Framework for Rank-based Evaluation Metrics for Link Prediction in Knowledge Graphs](https://mlsanity.com/paper/2203.07544/) (2022)
- [Comprehensive Analysis of Negative Sampling in Knowledge Graph Representation Learning](https://mlsanity.com/paper/2206.10140/) (2022)
- [Construction of Knowledge Graphs: State and Challenges](https://mlsanity.com/paper/2302.11509/) (2023)
- [Knowledge Sheaves: A Sheaf-Theoretic Framework for Knowledge Graph Embedding](https://mlsanity.com/paper/2110.03789/) (2021)
- [Implications of Topological Imbalance for Representation Learning on Biomedical Knowledge Graphs](https://mlsanity.com/paper/2112.06567/) (2021)
- [Improving Scene Graph Classification by Exploiting Knowledge from Texts](https://mlsanity.com/paper/2102.04760/) (2021)
- [Sem@K: Is my knowledge graph embedding model semantic-aware?](https://mlsanity.com/paper/2301.05601/) (2023)
- [Cognitive Semantic Communication Systems Driven by Knowledge Graph: Principle, Implementation, and Performance Evaluation](https://mlsanity.com/paper/2303.08546/) (2023)
- [Comprehensible Artificial Intelligence on Knowledge Graphs: A survey](https://mlsanity.com/paper/2404.03499/) (2024)
- [KGTuner: Efficient Hyper-parameter Search for Knowledge Graph Learning](https://mlsanity.com/paper/2205.02460/) (2022)
- [Re-thinking Knowledge Graph Completion Evaluation from an Information Retrieval Perspective](https://mlsanity.com/paper/2205.04105/) (2022)
- [A Retrieve-and-Read Framework for Knowledge Graph Link Prediction](https://mlsanity.com/paper/2212.09724/) (2022)
- [KGrEaT: A Framework to Evaluate Knowledge Graphs via Downstream Tasks](https://mlsanity.com/paper/2308.10537/) (2023)
- [EAGER: Embedding-Assisted Entity Resolution for Knowledge Graphs](https://mlsanity.com/paper/2101.06126/) (2021)
- [Knowledge Graphs Querying](https://mlsanity.com/paper/2305.14485/) (2023)
- [BioBLP: A Modular Framework for Learning on Multimodal Biomedical Knowledge Graphs](https://mlsanity.com/paper/2306.03606/) (2023)

## Abstract (arXiv)

The heterogeneity in recently published knowledge graph embedding models' implementations, training, and evaluation has made fair and thorough comparisons difficult. In order to assess the reproducibility of previously published results, we re-implemented and evaluated 21 interaction models in the PyKEEN software package. Here, we outline which results could be reproduced with their reported hyper-parameters, which could only be reproduced with alternate hyper-parameters, and which could not be reproduced at all as well as provide insight as to why this might be the case. We then performed a large-scale benchmarking on four datasets with several thousands of experiments and 24,804 GPU hours of computation time. We present insights gained as to best practices, best configurations for each model, and where improvements could be made over previously published best configurations. Our results highlight that the combination of model architecture, training approach, loss function, and the explicit modeling of inverse relations is crucial for a model's performances, and not only determined by the model architecture. We provide evidence that several architectures can obtain results competitive to the state-of-the-art when configured carefully. We have made all code, experimental configurations, results, and analyses that lead to our interpretations available at https://github.com/pykeen/pykeen and https://github.com/pykeen/benchmarking