# A Comprehensive Evaluation of Neural SPARQL Query Generation from Natural Language Questions

arXiv: https://arxiv.org/abs/2304.07772 · HTML: https://mlsanity.com/paper/2304.07772/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 49 references that are in the map.
- First posted on arXiv 16 Apr 2023.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [Convolutional Sequence to Sequence Learning](https://mlsanity.com/paper/1705.03122/) (2017)
- [Incorporating Copying Mechanism in Sequence-to-Sequence Learning](https://mlsanity.com/paper/1603.06393/) (2016)
- [SGPT: GPT Sentence Embeddings for Semantic Search](https://mlsanity.com/paper/2202.08904/) (2022)
- [Compositional generalization through meta sequence-to-sequence learning](https://mlsanity.com/paper/1906.05381/) (2019)
- [Modern Baselines for SPARQL Semantic Parsing](https://mlsanity.com/paper/2204.12793/) (2022)
- [Neural Machine Translating from Natural Language to SPARQL](https://mlsanity.com/paper/1906.09302/) (2019)
- [Leveraging Frequent Query Substructures to Generate Formal Queries for Complex Question Answering](https://mlsanity.com/paper/1908.11053/) (2019)
- [SPARQL as a Foreign Language](https://mlsanity.com/paper/1708.07624/) (2017)
- [Semantic Parsing Natural Language into SPARQL: Improving Target Language Representation with Neural Attention](https://mlsanity.com/paper/1803.04329/) (2018)
- [A Copy Mechanism for Handling Knowledge Base Elements in SPARQL Neural Machine Translation](https://mlsanity.com/paper/2211.10271/) (2022)
- [SPBERT: An Efficient Pre-training BERT on SPARQL Queries for Question Answering over Knowledge Graphs](https://mlsanity.com/paper/2106.09997/) (2021)

## Built on by (most cited first, 9 of 9)

- [Assessing SPARQL capabilities of Large Language Models](https://mlsanity.com/paper/2409.05925/) (2024)
- [Leveraging small language models for Text2SPARQL tasks to improve the resilience of AI assistance](https://mlsanity.com/paper/2405.17076/) (2024)
- [SPARQL Query Generation with LLMs: Measuring the Impact of Training Data Memorization and Knowledge Injection](https://mlsanity.com/paper/2507.13859/) (2025)
- [Reducing Hallucinations in Language Model-based SPARQL Query Generation Using Post-Generation Memory Retrieval](https://mlsanity.com/paper/2502.13369/) (2025)
- [Q-NL Verifier: Leveraging Synthetic Data for Robust Knowledge Graph Question Answering](https://mlsanity.com/paper/2503.01385/) (2025)
- [FRASE: Structured Representations for Generalizable SPARQL Query Generation](https://mlsanity.com/paper/2503.22144/) (2025)
- [InteracSPARQL: An Interactive System for SPARQL Query Refinement Using Natural Language Explanations](https://mlsanity.com/paper/2511.02002/) (2025)
- [Graph Query Generation with Constraint-guided Large Language Agents](https://mlsanity.com/paper/2605.00845/) (2026)
- [DeSQ: Decomposition-based SPARQL Query Generation](https://mlsanity.com/paper/2606.00203/) (2026)

## Abstract (arXiv)

In recent years, the field of neural machine translation (NMT) for SPARQL query generation has witnessed significant growth. Incorporating the copy mechanism with traditional encoder-decoder architectures and using pre-trained encoder-decoders and large language models have set new performance benchmarks. This paper presents various experiments that replicate and expand upon recent NMT-based SPARQL generation studies, comparing pre-trained language models (PLMs), non-pre-trained language models (NPLMs), and large language models (LLMs), highlighting the impact of question annotation and the copy mechanism and testing various fine-tuning methods using LLMs. In particular, we provide a systematic error analysis of the models and test their generalization ability. Our study demonstrates that the copy mechanism yields significant performance enhancements for most PLMs and NPLMs. Annotating the data is pivotal to generating correct URIs, with the "tag-within" strategy emerging as the most effective approach. Additionally, our findings reveal that the primary source of errors stems from incorrect URIs in SPARQL queries that are sometimes replaced with hallucinated URIs when using base models. This does not happen using the copy mechanism, but it sometimes leads to selecting wrong URIs among candidates. Finally, the performance of the tested LLMs fell short of achieving the desired outcomes.