# Benchmarking Large Language Models for Knowledge Graph Validation

arXiv: https://arxiv.org/abs/2602.10748 · HTML: https://mlsanity.com/paper/2602.10748/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge-graph and structured-knowledge RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 22 of its 72 references that are in the map.
- First posted on arXiv 11 Feb 2026.

## Placed in

- Retrieval-augmented generation / [Knowledge-graph and structured-knowledge RAG](https://mlsanity.com/rag/knowledge-graph-and-structured-knowledge-rag/)

## Builds on (most cited first, 22 of 22)

- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [A Survey on Knowledge Graphs: Representation, Acquisition and Applications](https://mlsanity.com/paper/2002.00388/) (2020)
- [Knowledge Graphs](https://mlsanity.com/paper/2003.02320/) (2020)
- [LLMs Get Lost In Multi-Turn Conversation](https://mlsanity.com/paper/2505.06120/) (2025)
- [Training Language Models to Self-Correct via Reinforcement Learning](https://mlsanity.com/paper/2409.12917/) (2024)
- [A Survey on Automated Fact-Checking](https://mlsanity.com/paper/2108.11896/) (2021)
- [When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs](https://mlsanity.com/paper/2406.01297/) (2024)
- [Head-to-Tail: How Knowledgeable are Large Language Models (LLMs)? A.K.A. Will LLMs Replace Knowledge Graphs?](https://mlsanity.com/paper/2308.10168/) (2023)
- [RumourEval 2019: Determining Rumour Veracity and Support for Rumours](https://mlsanity.com/paper/1809.06683/) (2018)
- [Reasoning Aware Self-Consistency: Leveraging Reasoning Paths for Efficient LLM Sampling](https://mlsanity.com/paper/2408.17017/) (2024)
- [FactKG: Fact Verification via Reasoning on Knowledge Graphs](https://mlsanity.com/paper/2305.06590/) (2023)
- [A Benchmark Dataset of Check-worthy Factual Claims](https://mlsanity.com/paper/2004.14425/) (2020)
- [RAGAR, Your Falsehood Radar: RAG-Augmented Reasoning for Political Fact-Checking using Multimodal Large Language Models](https://mlsanity.com/paper/2404.12065/) (2024)
- [Discriminative Predicate Path Mining for Fact Checking in Knowledge Graphs](https://mlsanity.com/paper/1510.05911/) (2015)
- [Contextual Document Embeddings](https://mlsanity.com/paper/2410.02525/) (2024)
- [Finding Streams in Knowledge Graphs to Support Fact Checking](https://mlsanity.com/paper/1708.07239/) (2017)
- [Can Language Models Act as Knowledge Bases at Scale?](https://mlsanity.com/paper/2402.14273/) (2024)
- [Efficient Knowledge Graph Accuracy Evaluation](https://mlsanity.com/paper/1907.09657/) (2019)
- [How Reliable are LLMs as Knowledge Bases? Re-thinking Facutality and Consistency](https://mlsanity.com/paper/2407.13578/) (2024)
- [KGValidator: A Framework for Automatic Validation of Knowledge Graph Construction](https://mlsanity.com/paper/2404.15923/) (2024)
- [Creating a Scholarly Knowledge Graph from Survey Article Tables](https://mlsanity.com/paper/2012.00456/) (2020)
- [Credible Intervals for Knowledge Graph Accuracy Estimation](https://mlsanity.com/paper/2502.18961/) (2025)

## Built on by (most cited first, 1 of 1)

- [Evaluation of Pipelines for Data Integration into Knowledge Graphs](https://mlsanity.com/paper/2605.22304/) (2026)

## Abstract (arXiv)

Knowledge Graphs (KGs) store structured factual knowledge by linking entities through relationships, crucial for many applications. These applications depend on the KG's factual accuracy, so verifying facts is essential, yet challenging. Expert manual verification is ideal but impractical on a large scale. Automated methods show promise but are not ready for real-world KGs. Large Language Models (LLMs) offer potential with their semantic understanding and knowledge access, yet their suitability and effectiveness for KG fact validation remain largely unexplored. In this paper, we introduce FactCheck, a benchmark designed to evaluate LLMs for KG fact validation across three key dimensions: (1) LLMs internal knowledge; (2) external evidence via Retrieval-Augmented Generation (RAG); and (3) aggregated knowledge employing a multi-model consensus strategy. We evaluated open-source and commercial LLMs on three diverse real-world KGs. FactCheck also includes a RAG dataset with 2+ million documents tailored for KG fact validation. Additionally, we offer an interactive exploration platform for analyzing verification decisions. The experimental analyses demonstrate that while LLMs yield promising results, they are still not sufficiently stable and reliable to be used in real-world KG validation scenarios. Integrating external evidence through RAG methods yields fluctuating performance, providing inconsistent improvements over more streamlined approaches -- at higher computational costs. Similarly, strategies based on multi-model consensus do not consistently outperform individual models, underscoring the lack of a one-fits-all solution. These findings further emphasize the need for a benchmark like FactCheck to systematically evaluate and drive progress on this difficult yet crucial task.