# Knowledge Graph Question Answering Datasets and Their Generalizability: Are They Enough for Future Research?

arXiv: https://arxiv.org/abs/2205.06573 · HTML: https://mlsanity.com/paper/2205.06573/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 43 references that are in the map.
- First posted on arXiv 13 May 2022.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [The Web as a Knowledge-base for Answering Complex Questions](https://mlsanity.com/paper/1803.06643/) (2018)
- [Measuring Compositional Generalization: A Comprehensive Method on Realistic Data](https://mlsanity.com/paper/1912.09713/) (2019)
- [Beyond I.I.D.: Three Levels of Generalization for Question Answering on Knowledge Bases](https://mlsanity.com/paper/2011.07743/) (2020)
- [Variational Reasoning for Question Answering with Knowledge Graph](https://mlsanity.com/paper/1709.04071/) (2017)
- [Large-scale Simple Question Answering with Memory Networks](https://mlsanity.com/paper/1506.02075/) (2015)
- [Question Answering Over Temporal Knowledge Graphs](https://mlsanity.com/paper/2106.01515/) (2021)
- [Complex Temporal Question Answering on Knowledge Graphs](https://mlsanity.com/paper/2109.08935/) (2021)
- [An Interpretable Reasoning Network for Multi-Relation Question Answering](https://mlsanity.com/paper/1801.04726/) (2018)
- [Generating Factoid Questions With Recurrent Neural Networks: The 30M Factoid Question-Answer Corpus](https://mlsanity.com/paper/1603.06807/) (2016)
- [A Comparative Survey of Recent Natural Language Interfaces for Databases](https://mlsanity.com/paper/1906.08990/) (2019)
- [Event-QA: A Dataset for Event-Centric Question Answering over Knowledge Graphs](https://mlsanity.com/paper/2004.11861/) (2020)
- [Outlining and Filling: Hierarchical Query Graph Generation for Answering Complex Questions over Knowledge Graphs](https://mlsanity.com/paper/2111.00732/) (2021)
- [Leveraging Frequent Query Substructures to Generate Formal Queries for Complex Question Answering](https://mlsanity.com/paper/1908.11053/) (2019)
- [RuBQ: A Russian Dataset for Question Answering over Wikidata](https://mlsanity.com/paper/2005.10659/) (2020)
- [Knowledge Graph Question Answering using Graph-Pattern Isomorphism](https://mlsanity.com/paper/2103.06752/) (2021)
- [No One is Perfect: Analysing the Performance of Question Answering Components over the DBpedia Knowledge Graph](https://mlsanity.com/paper/1809.10044/) (2018)

## Built on by (most cited first, 13 of 13)

- [Can LLMs Evaluate Complex Attribution in QA? Automatic Benchmarking using Knowledge Graphs](https://mlsanity.com/paper/2401.14640/) (2024)
- [Aligning Large Language Models to a Domain-specific Graph Database for NL2GQL](https://mlsanity.com/paper/2402.16567/) (2024)
- [Diagnosing and Addressing Pitfalls in KG-RAG Datasets: Toward More Reliable Benchmarking](https://mlsanity.com/paper/2505.23495/) (2025)
- [Beyond Isolation: Multi-Agent Synergy for Improving Knowledge Graph Construction](https://mlsanity.com/paper/2312.03022/) (2023)
- [The Lifecycle of "Facts": A Survey of Social Bias in Knowledge Graphs](https://mlsanity.com/paper/2210.03353/) (2022)
- [A Copy Mechanism for Handling Knowledge Base Elements in SPARQL Neural Machine Translation](https://mlsanity.com/paper/2211.10271/) (2022)
- [The Wikidata Query Logs Dataset](https://mlsanity.com/paper/2602.14594/) (2026)
- [Developing PUGG for Polish: A Modern Approach to KBQA, MRC, and IR Dataset Construction](https://mlsanity.com/paper/2408.02337/) (2024)
- [The benefits of query-based KGQA systems for complex and temporal questions in LLM era](https://mlsanity.com/paper/2507.11954/) (2025)
- [BioGraphletQA: Knowledge-Anchored Generation of Complex QA Datasets](https://mlsanity.com/paper/2604.26048/) (2026)
- [R2-CoD: Understanding Text-Graph Complementarity in Relational Reasoning via Knowledge Co-Distillation](https://mlsanity.com/paper/2508.01475/) (2025)
- [HOME-KGQA: A Benchmark Dataset for Multimodal Knowledge Graph Question Answering on Household Daily Activities](https://mlsanity.com/paper/2605.09348/) (2026)
- [AutoSchema: Live Schema Grounding for Agentic Text-to-Sparql over Heterogeneous Knowledge Graphs](https://mlsanity.com/paper/2608.14228/) (2026)

## Abstract (arXiv)

Existing approaches on Question Answering over Knowledge Graphs (KGQA) have weak generalizability. That is often due to the standard i.i.d. assumption on the underlying dataset. Recently, three levels of generalization for KGQA were defined, namely i.i.d., compositional, zero-shot. We analyze 25 well-known KGQA datasets for 5 different Knowledge Graphs (KGs). We show that according to this definition many existing and online available KGQA datasets are either not suited to train a generalizable KGQA system or that the datasets are based on discontinued and out-dated KGs. Generating new datasets is a costly process and, thus, is not an alternative to smaller research groups and companies. In this work, we propose a mitigation method for re-splitting available KGQA datasets to enable their applicability to evaluate generalization, without any cost and manual effort. We test our hypothesis on three KGQA datasets, i.e., LC-QuAD, LC-QuAD 2.0 and QALD-9). Experiments on re-splitted KGQA datasets demonstrate its effectiveness towards generalizability. The code and a unified way to access 18 available datasets is online at https://github.com/semantic-systems/KGQA-datasets as well as https://github.com/semantic-systems/KGQA-datasets-generalization.