mlsanityData to 7 Oct 2026

Developing a Scalable Benchmark for Assessing Large Language Models in Knowledge Graph Engineering

arXiv 2308.16622 ↗First posted 31 Aug 2023cs.AI cs.CL cs.DBcited by 13

Not on the map. Why?

Abstract (arXiv)

As the field of Large Language Models (LLMs) evolves at an accelerated pace, the critical need to assess and monitor their performance emerges. We introduce a benchmarking framework focused on knowledge graph engineering (KGE) accompanied by three challenges addressing syntax and error correction, facts extraction and dataset generation. We show that while being a useful tool, LLMs are yet unfit to assist in knowledge graph generation with zero-shot prompting. Consequently, our LLM-KG-Bench framework provides automatic evaluation and storage of LLM responses as well as statistical data and visualization tools to support tracking of prompt engineering and model performance.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 3 of its 5 references that are in the map. First posted on arXiv 31 Aug 2023.

Builds on

3 of 5 references.

References · 3

  1. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  2. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 663
  3. 2023LLM-assisted Knowledge Graph Engineering: Experiments with ChatGPTcited by 42

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 3; 2025: 7; 2026: 2.

13 papers.

Sort:

By area:

  1. 2024Graph Machine Learning in the Era of Large Language Models (LLMs)cited by 33
  2. 2025Large Language Models Meet Knowledge Graphs for Question Answering: Synthesis and Opportunitiescited by 30
  3. 2024Assessing SPARQL capabilities of Large Language Modelscited by 14
  4. 2024Leveraging small language models for Text2SPARQL tasks to improve the resilience of AI assistancecited by 13
  5. 2023Benchmarking the Abilities of Large Language Models for RDF Knowledge Graph Creation and Comprehension: How Well Do LLMs Speak Turtle?cited by 11
  6. 2025Retrieval-Augmented Generation of Ontologies from Relational Databasescited by 4
  7. 2025LLM-KG-Bench 3.0: A Compass for SemanticTechnology Capabilities in the Ocean of LLMscited by 3
  8. 2025ARUQULA -- An LLM based Text2SPARQL Approach using ReAct and Knowledge Graph Exploration Utilitiescited by 3
  9. 2025Domain Specific Benchmarks for Evaluating Multimodal Large Language Modelscited by 2
  10. 2025How do Scaling Laws Apply to Knowledge Graph Engineering Tasks? The Impact of Model Size on Large Language Model Performancecited by 1
  11. 2026Integrating Graphs, Large Language Models, and Agents: Reasoning and Retrievalcited by 1
  12. 2025Characterizing Knowledge Graph Tasks in LLM Benchmarks Using Cognitive Complexity Frameworkscited by 0
1 more
  1. 2026LLM-based Triplet Extraction from Financial Reportscited by 0