# KG-LLM-Bench: A Scalable Benchmark for Evaluating LLM Reasoning on Textualized Knowledge Graphs

arXiv: https://arxiv.org/abs/2504.07087 · HTML: https://mlsanity.com/paper/2504.07087/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning.
- Builds on 18 of its 52 references that are in the map.
- First posted on arXiv 9 Apr 2025.

## Placed in

- LLM reasoning / [Knowledge- and tool-augmented reasoning](https://mlsanity.com/lr/knowledge-and-tool-augmented-reasoning/)

## Builds on (most cited first, 18 of 18)

- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Can Language Models Solve Graph Problems in Natural Language?](https://mlsanity.com/paper/2305.10037/) (2023)
- [GPT4Graph: Can Large Language Models Understand Graph Structured Data ? An Empirical Evaluation and Benchmarking](https://mlsanity.com/paper/2305.15066/) (2023)
- [Talk like a Graph: Encoding Graphs for Large Language Models](https://mlsanity.com/paper/2310.04560/) (2023)
- [LLMs for Knowledge Graph Construction and Reasoning: Recent Capabilities and Future Opportunities](https://mlsanity.com/paper/2305.13168/) (2023)
- [A Survey of Graph Meets Large Language Model: Progress and Future Directions](https://mlsanity.com/paper/2311.12399/) (2023)
- [GraphArena: Evaluating and Exploring Large Language Models on Graph Computation](https://mlsanity.com/paper/2407.00379/) (2024)
- [Text-space Graph Foundation Models: Comprehensive Benchmarks and New Insights](https://mlsanity.com/paper/2406.10727/) (2024)
- [Graph Machine Learning in the Era of Large Language Models (LLMs)](https://mlsanity.com/paper/2404.14928/) (2024)
- [How Do Large Language Models Understand Graph Patterns? A Benchmark for Graph Pattern Comprehension](https://mlsanity.com/paper/2410.05298/) (2024)
- [Can Knowledge Graphs Make Large Language Models More Trustworthy? An Empirical Study Over Open-ended Question Answering](https://mlsanity.com/paper/2410.08085/) (2024)
- [Combining Knowledge Graphs and Large Language Models](https://mlsanity.com/paper/2407.06564/) (2024)
- [GraCoRe: Benchmarking Graph Comprehension and Complex Reasoning in Large Language Models](https://mlsanity.com/paper/2407.02936/) (2024)
- [Beyond Graphs: Can Large Language Models Comprehend Hypergraphs?](https://mlsanity.com/paper/2410.10083/) (2024)
- [CLR-Fact: Evaluating the Complex Logical Reasoning Capability of Large Language Models over Factual Knowledge](https://mlsanity.com/paper/2407.20564/) (2024)
- [Causal Reasoning in Large Language Models: A Knowledge Graph Approach](https://mlsanity.com/paper/2410.11588/) (2024)
- [WikiDataSets: Standardized sub-graphs from Wikidata](https://mlsanity.com/paper/1906.04536/) (2019)
- [G2T-LLM: Graph-to-Tree Text Encoding for Molecule Generation with Fine-Tuned Large Language Models](https://mlsanity.com/paper/2410.02198/) (2024)

## Built on by (most cited first, 6 of 6)

- [Knowledge-Grounded Agentic Large Language Models for Multi-Hazard Understanding from Reconnaissance Reports](https://mlsanity.com/paper/2511.14010/) (2025)
- [Knowledge Graph Representations for LLM-Based Policy Compliance Reasoning](https://mlsanity.com/paper/2604.27713/) (2026)
- [Evaluating Implicit Biases in LLM Reasoning through Logic Grid Puzzles](https://mlsanity.com/paper/2511.06160/) (2025)
- [MHGraphBench: Knowledge Graph-Grounded Benchmarking of Mental Health Knowledge in Large Language Models](https://mlsanity.com/paper/2605.15589/) (2026)
- [Causal-Audit: Explicit and Auditable Graph-based Reasoning via Target-Aware Causal Chain Construction](https://mlsanity.com/paper/2607.15281/) (2026)
- [GTA: Graph Theory Agent and Benchmark for Algorithmic Graph Reasoning with LLMs](https://mlsanity.com/paper/2609.12265/) (2026)

## Abstract (arXiv)

Knowledge graphs have emerged as a popular method for injecting up-to-date, factual knowledge into large language models (LLMs). This is typically achieved by converting the knowledge graph into text that the LLM can process in context. While multiple methods of encoding knowledge graphs have been proposed, the impact of this textualization process on LLM performance remains under-explored. We introduce KG-LLM-Bench, a comprehensive and extensible benchmark spanning five knowledge graph understanding tasks, and evaluate how different encoding strategies affect performance across various base models. Our extensive experiments with seven language models and five textualization strategies provide insights for optimizing LLM performance on KG reasoning tasks.