# KGrEaT: A Framework to Evaluate Knowledge Graphs via Downstream Tasks

arXiv: https://arxiv.org/abs/2308.10537 · HTML: https://mlsanity.com/paper/2308.10537/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 37 references that are in the map.
- First posted on arXiv 21 Aug 2023.

## Builds on (most cited first, 15 of 15)

- [Embedding Entities and Relations for Learning and Inference in Knowledge Bases](https://mlsanity.com/paper/1412.6575/) (2014)
- [Complex Embeddings for Simple Link Prediction](https://mlsanity.com/paper/1606.06357/) (2016)
- [Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs](https://mlsanity.com/paper/1603.09320/) (2016)
- [Bringing Light Into the Dark: A Large-scale Evaluation of Knowledge Graph Embedding Models Under a Unified Framework](https://mlsanity.com/paper/2006.13365/) (2020)
- [DGL-KE: Training Knowledge Graph Embeddings at Scale](https://mlsanity.com/paper/2004.08532/) (2020)
- [Realistic Re-evaluation of Knowledge Graph Completion Methods: An Experimental Study](https://mlsanity.com/paper/2003.08001/) (2020)
- [Knowledge Graphs on the Web -- an Overview](https://mlsanity.com/paper/2003.00719/) (2020)
- [RDF2Vec Light -- A Lightweight Approach for Knowledge Graph Embeddings](https://mlsanity.com/paper/2009.07659/) (2020)
- [Efficient Knowledge Graph Accuracy Evaluation](https://mlsanity.com/paper/1907.09657/) (2019)
- [Uncovering the Semantics of Wikipedia Categories](https://mlsanity.com/paper/1906.12089/) (2019)
- [Which Knowledge Graph Is Best for Me?](https://mlsanity.com/paper/1809.11099/) (2018)
- [The CaLiGraph Ontology as a Challenge for OWL Reasoners](https://mlsanity.com/paper/2110.05028/) (2021)
- [NASTyLinker: NIL-Aware Scalable Transformer-based Entity Linker](https://mlsanity.com/paper/2303.04426/) (2023)
- [Towards Knowledge Graphs Validation through Weighted Knowledge Sources](https://mlsanity.com/paper/2104.12622/) (2021)
- [DBkWik++ -- Multi Source Matching of Knowledge Graphs](https://mlsanity.com/paper/2210.02864/) (2022)

## Built on by (most cited first, 6 of 6)

- [KGGen: Extracting Knowledge Graphs from Plain Text with Language Models](https://mlsanity.com/paper/2502.09956/) (2025)
- [On Large-scale Evaluation of Embedding Models for Knowledge Graph Completion](https://mlsanity.com/paper/2504.08970/) (2025)
- [Are Large Language Models Effective Knowledge Graph Constructors?](https://mlsanity.com/paper/2510.11297/) (2025)
- [Evaluation of Pipelines for Data Integration into Knowledge Graphs](https://mlsanity.com/paper/2605.22304/) (2026)
- [A Unified Benchmark for Evaluating Knowledge Graph Construction Methods and Graph Neural Networks](https://mlsanity.com/paper/2605.05476/) (2026)
- [WiCleanData: Guaranteeing the Type Consistency of Wikidata by Taxonomy Refinement and Constraint Enforcement](https://mlsanity.com/paper/2609.20057/) (2026)

## Abstract (arXiv)

In recent years, countless research papers have addressed the topics of knowledge graph creation, extension, or completion in order to create knowledge graphs that are larger, more correct, or more diverse. This research is typically motivated by the argumentation that using such enhanced knowledge graphs to solve downstream tasks will improve performance. Nonetheless, this is hardly ever evaluated. Instead, the predominant evaluation metrics - aiming at correctness and completeness - are undoubtedly valuable but fail to capture the complete picture, i.e., how useful the created or enhanced knowledge graph actually is. Further, the accessibility of such a knowledge graph is rarely considered (e.g., whether it contains expressive labels, descriptions, and sufficient context information to link textual mentions to the entities of the knowledge graph). To better judge how well knowledge graphs perform on actual tasks, we present KGrEaT - a framework to estimate the quality of knowledge graphs via actual downstream tasks like classification, clustering, or recommendation. Instead of comparing different methods of processing knowledge graphs with respect to a single task, the purpose of KGrEaT is to compare various knowledge graphs as such by evaluating them on a fixed task setup. The framework takes a knowledge graph as input, automatically maps it to the datasets to be evaluated on, and computes performance metrics for the defined tasks. It is built in a modular way to be easily extendable with additional tasks and datasets.