# Characterizing Knowledge Graph Tasks in LLM Benchmarks Using Cognitive Complexity Frameworks

arXiv: https://arxiv.org/abs/2509.19347 · HTML: https://mlsanity.com/paper/2509.19347/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 6 of its 11 references that are in the map.
- First posted on arXiv 17 Sep 2025.

## Builds on (most cited first, 6 of 6)

- [Unifying Large Language Models and Knowledge Graphs: A Roadmap](https://mlsanity.com/paper/2306.08302/) (2023)
- [LLM-assisted Knowledge Graph Engineering: Experiments with ChatGPT](https://mlsanity.com/paper/2307.06917/) (2023)
- [Assessing SPARQL capabilities of Large Language Models](https://mlsanity.com/paper/2409.05925/) (2024)
- [Developing a Scalable Benchmark for Assessing Large Language Models in Knowledge Graph Engineering](https://mlsanity.com/paper/2308.16622/) (2023)
- [Benchmarking the Abilities of Large Language Models for RDF Knowledge Graph Creation and Comprehension: How Well Do LLMs Speak Turtle?](https://mlsanity.com/paper/2309.17122/) (2023)
- [LLM-KG-Bench 3.0: A Compass for SemanticTechnology Capabilities in the Ocean of LLMs](https://mlsanity.com/paper/2505.13098/) (2025)

## Abstract (arXiv)

Large Language Models (LLMs) are increasingly used for tasks involving Knowledge Graphs (KGs), whose evaluation typically focuses on accuracy and output correctness. We propose a complementary task characterization approach using three complexity frameworks from cognitive psychology. Applying this to the LLM-KG-Bench framework, we highlight value distributions, identify underrepresented demands and motivate richer interpretation and diversity for benchmark evaluation tasks.