# Assessing SPARQL capabilities of Large Language Models

arXiv: https://arxiv.org/abs/2409.05925 · HTML: https://mlsanity.com/paper/2409.05925/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 36 references that are in the map.
- First posted on arXiv 9 Sep 2024.

## Builds on (most cited first, 15 of 15)

- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [Unifying Large Language Models and Knowledge Graphs: A Roadmap](https://mlsanity.com/paper/2306.08302/) (2023)
- [Knowledge Graphs](https://mlsanity.com/paper/2003.02320/) (2020)
- [Large-scale Simple Question Answering with Memory Networks](https://mlsanity.com/paper/1506.02075/) (2015)
- [FlexKBQA: A Flexible LLM-Powered Framework for Few-Shot Knowledge Base Question Answering](https://mlsanity.com/paper/2308.12060/) (2023)
- [LLM-assisted Knowledge Graph Engineering: Experiments with ChatGPT](https://mlsanity.com/paper/2307.06917/) (2023)
- [Modern Baselines for SPARQL Semantic Parsing](https://mlsanity.com/paper/2204.12793/) (2022)
- [SPARQL Generation: an analysis on fine-tuning OpenLLaMA for Question Answering over a Life Science Knowledge Graph](https://mlsanity.com/paper/2402.04627/) (2024)
- [Developing a Scalable Benchmark for Assessing Large Language Models in Knowledge Graph Engineering](https://mlsanity.com/paper/2308.16622/) (2023)
- [Leveraging small language models for Text2SPARQL tasks to improve the resilience of AI assistance](https://mlsanity.com/paper/2405.17076/) (2024)
- [Benchmarking the Abilities of Large Language Models for RDF Knowledge Graph Creation and Comprehension: How Well Do LLMs Speak Turtle?](https://mlsanity.com/paper/2309.17122/) (2023)
- [A Comprehensive Evaluation of Neural SPARQL Query Generation from Natural Language Questions](https://mlsanity.com/paper/2304.07772/) (2023)
- [SPARQL Generation with Entity Pre-trained GPT for KG Question Answering](https://mlsanity.com/paper/2402.00969/) (2024)
- [A Copy Mechanism for Handling Knowledge Base Elements in SPARQL Neural Machine Translation](https://mlsanity.com/paper/2211.10271/) (2022)

## Built on by (most cited first, 14 of 14)

- [LLM-based SPARQL Query Generation from Natural Language over Federated Knowledge Graphs](https://mlsanity.com/paper/2410.06062/) (2024)
- [Enhancing Text2Cypher with Schema Filtering](https://mlsanity.com/paper/2505.05118/) (2025)
- [SPARQL Query Generation with LLMs: Measuring the Impact of Training Data Memorization and Knowledge Injection](https://mlsanity.com/paper/2507.13859/) (2025)
- [LLM-KG-Bench 3.0: A Compass for SemanticTechnology Capabilities in the Ocean of LLMs](https://mlsanity.com/paper/2505.13098/) (2025)
- [ARUQULA -- An LLM based Text2SPARQL Approach using ReAct and Knowledge Graph Exploration Utilities](https://mlsanity.com/paper/2510.02200/) (2025)
- [OntologyRAG: Better and Faster Biomedical Code Mapping with Retrieval-Augmented Generation (RAG) Leveraging Ontology Knowledge Graphs and Large Language Models](https://mlsanity.com/paper/2502.18992/) (2025)
- [Bridging Dual Knowledge Graphs for Multi-Hop Question Answering in Construction Safety](https://mlsanity.com/paper/2507.13625/) (2025)
- [RAGONITE: Iterative Retrieval on Induced Databases and Verbalized RDF for Conversational QA over KGs with RAG](https://mlsanity.com/paper/2412.17690/) (2024)
- [How do Scaling Laws Apply to Knowledge Graph Engineering Tasks? The Impact of Model Size on Large Language Model Performance](https://mlsanity.com/paper/2505.16276/) (2025)
- [GraphSeek: Next-Generation Graph Analytics with LLMs](https://mlsanity.com/paper/2602.11052/) (2026)
- [Knowledge Conceptualization Impacts RAG Efficacy](https://mlsanity.com/paper/2507.09389/) (2025)
- [Characterizing Knowledge Graph Tasks in LLM Benchmarks Using Cognitive Complexity Frameworks](https://mlsanity.com/paper/2509.19347/) (2025)
- [MetaboT: An LLM-based Multi-Agent Frameworkfor Interactive Analysis of Mass SpectrometryMetabolomics Knowledge Graphs](https://mlsanity.com/paper/2510.01724/) (2025)
- [Enabling Knowledge Graph Understanding at Scale with the EXplore Your Graphs ENgine (EXYGEN)](https://mlsanity.com/paper/2609.11569/) (2026)

## Abstract (arXiv)

The integration of Large Language Models (LLMs) with Knowledge Graphs (KGs) offers significant synergistic potential for knowledge-driven applications. One possible integration is the interpretation and generation of formal languages, such as those used in the Semantic Web, with SPARQL being a core technology for accessing KGs. In this paper, we focus on measuring out-of-the box capabilities of LLMs to work with SPARQL and more specifically with SPARQL SELECT queries applying a quantitative approach. We implemented various benchmarking tasks in the LLM-KG-Bench framework for automated execution and evaluation with several LLMs. The tasks assess capabilities along the dimensions of syntax, semantic read, semantic create, and the role of knowledge graph prompt inclusion. With this new benchmarking tasks, we evaluated a selection of GPT, Gemini, and Claude models. Our findings indicate that working with SPARQL SELECT queries is still challenging for LLMs and heavily depends on the specific LLM as well as the complexity of the task. While fixing basic syntax errors seems to pose no problems for the best of the current LLMs evaluated, creating semantically correct SPARQL SELECT queries is difficult in several cases.