# FIRESPARQL: A LLM-based Framework for SPARQL Query Generation over Scholarly Knowledge Graphs

arXiv: https://arxiv.org/abs/2508.10467 · HTML: https://mlsanity.com/paper/2508.10467/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 29 references that are in the map.
- First posted on arXiv 14 Aug 2025.

## Builds on (most cited first, 13 of 13)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering](https://mlsanity.com/paper/2104.06378/) (2021)
- [Knowledge Graph Prompting for Multi-Document Question Answering](https://mlsanity.com/paper/2308.11730/) (2023)
- [Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG](https://mlsanity.com/paper/2410.05983/) (2024)
- [Sufficient Context: A New Lens on Retrieval Augmented Generation Systems](https://mlsanity.com/paper/2411.06037/) (2024)
- [ChatGPT versus Traditional Question Answering for Knowledge Graphs: Current Status and Future Directions Towards Knowledge Graph Chatbots](https://mlsanity.com/paper/2302.06466/) (2023)
- [QALD-9-plus: A Multilingual Dataset for Question Answering over DBpedia and Wikidata Translated by Native Speakers](https://mlsanity.com/paper/2202.00120/) (2022)
- [Leveraging LLMs in Scholarly Knowledge Graph Question Answering](https://mlsanity.com/paper/2311.09841/) (2023)
- [SPARQL Generation with Entity Pre-trained GPT for KG Question Answering](https://mlsanity.com/paper/2402.00969/) (2024)
- [TechGPT-2.0: A large language model project to solve the task of knowledge graph construction](https://mlsanity.com/paper/2401.04507/) (2024)
- [Knowledge AI: Fine-tuning NLP Models for Facilitating Scientific Knowledge Extraction and Understanding](https://mlsanity.com/paper/2408.04651/) (2024)

## Built on by (most cited first, 5 of 5)

- [Performance Evaluation of LLMs in Automated RDF Knowledge Graph Generation](https://mlsanity.com/paper/2603.29878/) (2026)
- [MARS: Multi-hop Adaptive Retrieval and SPARQL Generation for KGQA](https://mlsanity.com/paper/2607.14561/) (2026)
- [GGC: Selective Query Correction for Reliable Text-to-SPARQL Generation](https://mlsanity.com/paper/2607.28082/) (2026)
- [Enabling Knowledge Graph Understanding at Scale with the EXplore Your Graphs ENgine (EXYGEN)](https://mlsanity.com/paper/2609.11569/) (2026)
- [Natural Language Knowledge Graph Query Execution: Leveraging Controlled Semantics in the LLM Context Window](https://mlsanity.com/paper/2609.14652/) (2026)

## Abstract (arXiv)

Question answering over Scholarly Knowledge Graphs (SKGs) remains a challenging task due to the complexity of scholarly content and the intricate structure of these graphs. Large Language Model (LLM) approaches could be used to translate natural language questions (NLQs) into SPARQL queries; however, these LLM-based approaches struggle with SPARQL query generation due to limited exposure to SKG-specific content and the underlying schema. We identified two main types of errors in the LLM-generated SPARQL queries: (i) structural inconsistencies, such as missing or redundant triples in the queries, and (ii) semantic inaccuracies, where incorrect entities or properties are shown in the queries despite a correct query structure. To address these issues, we propose FIRESPARQL, a modular framework that supports fine-tuned LLMs as a core component, with optional context provided via retrieval-augmented generation (RAG) and a SPARQL query correction layer. We evaluate the framework on the SciQA Benchmark using various configurations (zero-shot, zero-shot with RAG, one-shot, fine-tuning, and fine-tuning with RAG) and compare the performance with baseline and state-of-the-art approaches. We measure query accuracy using BLEU and ROUGE metrics, and query result accuracy using relaxed exact match(RelaxedEM), with respect to the gold standards containing the NLQs, SPARQL queries, and the results of the queries. Experimental results demonstrate that fine-tuning achieves the highest overall performance, reaching 0.90 ROUGE-L for query accuracy and 0.85 RelaxedEM for result accuracy on the test set.