# Systematic Evaluation of Knowledge Graph Repair with Large Language Models

arXiv: https://arxiv.org/abs/2507.22419 · HTML: https://mlsanity.com/paper/2507.22419/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 46 references that are in the map.
- First posted on arXiv 30 Jul 2025.

## Builds on (most cited first, 12 of 12)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Large Language Models as General Pattern Machines](https://mlsanity.com/paper/2307.04721/) (2023)
- [Retrieval-Augmented Generation with Knowledge Graphs for Customer Service Question Answering](https://mlsanity.com/paper/2404.17723/) (2024)
- [SimKGC: Simple Contrastive Knowledge Graph Completion with Pre-trained Language Models](https://mlsanity.com/paper/2203.02167/) (2022)
- [Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wisely](https://mlsanity.com/paper/2409.14924/) (2024)
- [Commonsense Knowledge Base Completion with Structural and Semantic Context](https://mlsanity.com/paper/1910.02915/) (2019)
- [Text2KGBench: A Benchmark for Ontology-Driven Knowledge Graph Generation from Text](https://mlsanity.com/paper/2308.02357/) (2023)
- [YAGO 4.5: A Large and Clean Knowledge Base with a Rich Taxonomy](https://mlsanity.com/paper/2308.11884/) (2023)
- [CodeKGC: Code Language Model for Generative Knowledge Graph Construction](https://mlsanity.com/paper/2304.09048/) (2023)
- [Knowledge Graph Generation From Text](https://mlsanity.com/paper/2211.10511/) (2022)
- [A Review of SHACL: From Data Validation to Schema Reasoning for RDF Graphs](https://mlsanity.com/paper/2112.01441/) (2021)
- [Trav-SHACL: Efficiently Validating Networks of SHACL Constraints](https://mlsanity.com/paper/2101.07136/) (2021)

## Abstract (arXiv)

We present a systematic approach for evaluating the quality of knowledge graph repairs with respect to constraint violations defined in shapes constraint language (SHACL). Current evaluation methods rely on ad hoc datasets, which limits the rigorous analysis of repair systems in more general settings. Our method addresses this gap by systematically generating violations using a novel mechanism, termed violation-inducing operations (VIOs). We use the proposed evaluation framework to assess a range of repair systems which we build using large language models. We analyze the performance of these systems across different prompting strategies. Results indicate that concise prompts containing both the relevant violated SHACL constraints and key contextual information from the knowledge graph yield the best performance.