# SPINACH: SPARQL-Based Information Navigation for Challenging Real-World Questions

arXiv: https://arxiv.org/abs/2407.11417 · HTML: https://mlsanity.com/paper/2407.11417/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 82 references that are in the map.
- First posted on arXiv 16 Jul 2024.

## Builds on (most cited first, 30 of 34)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories](https://mlsanity.com/paper/2212.10511/) (2022)
- [The Web as a Knowledge-base for Answering Complex Questions](https://mlsanity.com/paper/1803.06643/) (2018)
- [Measuring Compositional Generalization: A Comprehensive Method on Realistic Data](https://mlsanity.com/paper/1912.09713/) (2019)
- [Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graph](https://mlsanity.com/paper/2307.07697/) (2023)
- [MultiWOZ 2.2 : A Dialogue Dataset with Additional Annotation Corrections and State Tracking Baselines](https://mlsanity.com/paper/2007.12720/) (2020)
- [PullNet: Open Domain Question Answering with Iterative Retrieval on Knowledge Bases and Text](https://mlsanity.com/paper/1904.09537/) (2019)
- [Beyond I.I.D.: Three Levels of Generalization for Question Answering on Knowledge Bases](https://mlsanity.com/paper/2011.07743/) (2020)
- [Open Domain Question Answering Using Early Fusion of Knowledge Bases and Text](https://mlsanity.com/paper/1809.00782/) (2018)
- [With Little Power Comes Great Responsibility](https://mlsanity.com/paper/2010.06595/) (2020)
- [Improving Text-to-SQL Evaluation Methodology](https://mlsanity.com/paper/1806.09029/) (2018)
- [Large-scale Simple Question Answering with Memory Networks](https://mlsanity.com/paper/1506.02075/) (2015)
- [Case-based Reasoning for Natural Language Queries over Knowledge Bases](https://mlsanity.com/paper/2104.08762/) (2021)
- [RnG-KBQA: Generation Augmented Iterative Ranking for Knowledge Base Question Answering](https://mlsanity.com/paper/2109.08678/) (2021)
- [DecAF: Joint Decoding of Answers and Logical Forms for Question Answering over Knowledge Bases](https://mlsanity.com/paper/2210.00063/) (2022)
- [KQA Pro: A Dataset with Explicit Compositional Programs for Complex Question Answering over Knowledge Base](https://mlsanity.com/paper/2007.03875/) (2020)
- [ReFinED: An Efficient Zero-shot-capable Approach to End-to-End Entity Linking](https://mlsanity.com/paper/2207.04108/) (2022)
- [Can ChatGPT Replace Traditional KBQA Models? An In-depth Analysis of the Question Answering Performance of the GPT LLM Family](https://mlsanity.com/paper/2303.07992/) (2023)
- [WikiChat: Stopping the Hallucination of Large Language Model Chatbots by Few-Shot Grounding on Wikipedia](https://mlsanity.com/paper/2305.14292/) (2023)
- [ArcaneQA: Dynamic Program Induction and Contextualized Encoding for Knowledge Base Question Answering](https://mlsanity.com/paper/2204.08109/) (2022)
- [QALD-9-plus: A Multilingual Dataset for Question Answering over DBpedia and Wikidata Translated by Native Speakers](https://mlsanity.com/paper/2202.00120/) (2022)
- [A Study of the Quality of Wikidata](https://mlsanity.com/paper/2107.00156/) (2021)
- [Program Transfer for Answering Complex Questions over Knowledge Bases](https://mlsanity.com/paper/2110.05743/) (2021)
- [Modeling Semantics with Gated Graph Neural Networks for Knowledge Base Question Answering](https://mlsanity.com/paper/1808.04126/) (2018)
- [Finding needles in a haystack: Sampling Structurally-diverse Training Sets from Synthetic Data for Compositional Generalization](https://mlsanity.com/paper/2109.02575/) (2021)
- [RuBQ: A Russian Dataset for Question Answering over Wikidata](https://mlsanity.com/paper/2005.10659/) (2020)
- [Expanding End-to-End Question Answering on Differentiable Knowledge Graphs with Intersection](https://mlsanity.com/paper/2109.05808/) (2021)

## Built on by (most cited first, 27 of 27)

- [Multi-Agent Systems Execute Arbitrary Malicious Code](https://mlsanity.com/paper/2503.12188/) (2025)
- [CypherBench: Towards Precise Retrieval over Full-scale Modern Knowledge Graphs in the LLM Era](https://mlsanity.com/paper/2412.18702/) (2024)
- [DataSTORM: Deep Research on Large-Scale Databases using Exploratory Data Analysis and Data Storytelling](https://mlsanity.com/paper/2604.06474/) (2026)
- [SPARQL Query Generation with LLMs: Measuring the Impact of Training Data Memorization and Knowledge Injection](https://mlsanity.com/paper/2507.13859/) (2025)
- [Search-on-Graph: Iterative Informed Navigation for Large Language Model Reasoning on Knowledge Graphs](https://mlsanity.com/paper/2510.08825/) (2025)
- [ARUQULA -- An LLM based Text2SPARQL Approach using ReAct and Knowledge Graph Exploration Utilities](https://mlsanity.com/paper/2510.02200/) (2025)
- [SPARQL-LLM: Real-Time SPARQL Query Generation from Natural Language Questions](https://mlsanity.com/paper/2512.14277/) (2025)
- [Text2GraphQuery-Bench: A Text to Graph Query Benchmark](https://mlsanity.com/paper/2602.11745/) (2026)
- [LEMONADE: A Large Multilingual Expert-Annotated Abstractive Event Dataset for the Real World](https://mlsanity.com/paper/2506.00980/) (2025)
- [InteracSPARQL: An Interactive System for SPARQL Query Refinement Using Natural Language Explanations](https://mlsanity.com/paper/2511.02002/) (2025)
- [Self-Correction Distillation for Structured Data Question Answering](https://mlsanity.com/paper/2511.07998/) (2025)
- [When Should LLMs Be Less Specific? Selective Abstraction for Reliable Long-Form Text Generation](https://mlsanity.com/paper/2602.11908/) (2026)
- [The Wikidata Query Logs Dataset](https://mlsanity.com/paper/2602.14594/) (2026)
- [The benefits of query-based KGQA systems for complex and temporal questions in LLM era](https://mlsanity.com/paper/2507.11954/) (2025)
- [From Graph Retrieval to Schema Realization: Counterfactual Validation for Text-to-SPARQL over Heterogeneous Knowledge Graphs](https://mlsanity.com/paper/2508.01815/) (2025)
- [GRISP: Guided Recurrent IRI Selection over SPARQL Skeletons](https://mlsanity.com/paper/2604.21133/) (2026)
- [Question-to-Question Retrieval for Hallucination-Free Knowledge Access: An Approach for Wikipedia and Wikidata Question Answering](https://mlsanity.com/paper/2501.11301/) (2025)
- [SCRIBES: Web-Scale Script-Based Semi-Structured Data Extraction with Reinforcement Learning](https://mlsanity.com/paper/2510.01832/) (2025)
- [DeSQ: Decomposition-based SPARQL Query Generation](https://mlsanity.com/paper/2606.00203/) (2026)
- [Reducing Hallucinations in Complex Question Answering using Simple Graph-based Retrieval-Augmented Generation (long version)](https://mlsanity.com/paper/2606.05901/) (2026)
- [MARS: Multi-hop Adaptive Retrieval and SPARQL Generation for KGQA](https://mlsanity.com/paper/2607.14561/) (2026)
- [Making Mathematical Knowledge Explainable, Accessible and Interoperable Through Large Language Model Integration](https://mlsanity.com/paper/2607.24512/) (2026)
- [Detecting Knowledge Inconsistencies Across Text, Tables, and Knowledge Graphs](https://mlsanity.com/paper/2607.25959/) (2026)
- [KGVoyager: Knowledge Graph Agnostic Question Answering via Agentic Navigation](https://mlsanity.com/paper/2609.01780/) (2026)
- [Natural Language Knowledge Graph Query Execution: Leveraging Controlled Semantics in the LLM Context Window](https://mlsanity.com/paper/2609.14652/) (2026)
- [Symbolic Separation: Grounding Deep Agents in Knowledge Graphs for Trustworthy Operational Data Analytics](https://mlsanity.com/paper/2609.17107/) (2026)
- [CypherTurn: A Multi-Turn Benchmark for Conversational Text-to-Cypher Evaluation and the Autonomy Divergence](https://mlsanity.com/paper/2609.36987/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have led to significant improvements in the Knowledge Base Question Answering (KBQA) task. However, datasets used in KBQA studies do not capture the true complexity of KBQA tasks. They either have simple questions, use synthetically generated logical forms, or are based on small knowledge base (KB) schemas. We introduce the SPINACH dataset, an expert-annotated KBQA dataset collected from discussions on Wikidata's "Request a Query" forum with 320 decontextualized question-SPARQL pairs. The complexity of these in-the-wild queries calls for a KBQA system that can dynamically explore large and often incomplete schemas and reason about them, as it is infeasible to create a comprehensive training dataset. We also introduce an in-context learning KBQA agent, also called SPINACH, that mimics how a human expert would write SPARQLs to handle challenging questions. SPINACH achieves a new state of the art on the QALD-7, QALD-9 Plus and QALD-10 datasets by 31.0%, 27.0%, and 10.0% in F1, respectively, and coming within 1.6% of the fine-tuned LLaMA SOTA model on WikiWebQuestions. On our new SPINACH dataset, the SPINACH agent outperforms all baselines, including the best GPT-4-based KBQA agent, by at least 38.1% in F1.