# Knowledge Graph Question Answering Leaderboard: A Community Resource to Prevent a Replication Crisis arXiv: https://arxiv.org/abs/2201.08174 ยท HTML: https://mlsanity.com/paper/2201.08174/ ## Key facts - Cited by 14 papers in the map (data through 14 Oct 2026). - Builds on 24 of its 61 references that are in the map. - First posted on arXiv 20 Jan 2022. ## Builds on (most cited first, 24 of 24) - [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016) - [Open Graph Benchmark: Datasets for Machine Learning on Graphs](https://mlsanity.com/paper/2005.00687/) (2020) - [The Web as a Knowledge-base for Answering Complex Questions](https://mlsanity.com/paper/1803.06643/) (2018) - [Measuring Compositional Generalization: A Comprehensive Method on Realistic Data](https://mlsanity.com/paper/1912.09713/) (2019) - [Beyond I.I.D.: Three Levels of Generalization for Question Answering on Knowledge Bases](https://mlsanity.com/paper/2011.07743/) (2020) - [Variational Reasoning for Question Answering with Knowledge Graph](https://mlsanity.com/paper/1709.04071/) (2017) - [Large-scale Simple Question Answering with Memory Networks](https://mlsanity.com/paper/1506.02075/) (2015) - [Question Answering Over Temporal Knowledge Graphs](https://mlsanity.com/paper/2106.01515/) (2021) - [Complex Temporal Question Answering on Knowledge Graphs](https://mlsanity.com/paper/2109.08935/) (2021) - [An Interpretable Reasoning Network for Multi-Relation Question Answering](https://mlsanity.com/paper/1801.04726/) (2018) - [A Survey on Complex Question Answering over Knowledge Base: Recent Advances and Challenges](https://mlsanity.com/paper/2007.13069/) (2020) - [Generating Factoid Questions With Recurrent Neural Networks: The 30M Factoid Question-Answer Corpus](https://mlsanity.com/paper/1603.06807/) (2016) - [Utility is in the Eye of the User: A Critique of NLP Leaderboards](https://mlsanity.com/paper/2009.13888/) (2020) - [Event-QA: A Dataset for Event-Centric Question Answering over Knowledge Graphs](https://mlsanity.com/paper/2004.11861/) (2020) - [Message Passing for Complex Question Answering over Knowledge Graphs](https://mlsanity.com/paper/1908.06917/) (2019) - [Neural Machine Translating from Natural Language to SPARQL](https://mlsanity.com/paper/1906.09302/) (2019) - [EARL: Joint Entity and Relation Linking for Question Answering over Knowledge Graphs](https://mlsanity.com/paper/1801.03825/) (2018) - [Learning to Rank Query Graphs for Complex Question Answering over Knowledge Graphs](https://mlsanity.com/paper/1811.01118/) (2018) - [Towards a Question Answering System over the Semantic Web](https://mlsanity.com/paper/1803.00832/) (2018) - [RuBQ: A Russian Dataset for Question Answering over Wikidata](https://mlsanity.com/paper/2005.10659/) (2020) - [Question Answering over Knowledge Graphs via Structural Query Patterns](https://mlsanity.com/paper/1910.09760/) (2019) - [A Chinese Multi-type Complex Questions Answering Dataset over Wikidata](https://mlsanity.com/paper/2111.06086/) (2021) - [Modeling Global Semantics for Question Answering over Knowledge Bases](https://mlsanity.com/paper/2101.01510/) (2021) - [CBench: Towards Better Evaluation of Question Answering Over Knowledge Graphs](https://mlsanity.com/paper/2105.00811/) (2021) ## Built on by (most cited first, 14 of 14) - [Can ChatGPT Replace Traditional KBQA Models? An In-depth Analysis of the Question Answering Performance of the GPT LLM Family](https://mlsanity.com/paper/2303.07992/) (2023) - [Modern Baselines for SPARQL Semantic Parsing](https://mlsanity.com/paper/2204.12793/) (2022) - [Knowledge Base Question Answering: A Semantic Parsing Perspective](https://mlsanity.com/paper/2209.04994/) (2022) - [DBLP-QuAD: A Question Answering Dataset over the DBLP Scholarly Knowledge Graph](https://mlsanity.com/paper/2303.13351/) (2023) - [RAS: Retrieval-And-Structuring for Knowledge-Intensive LLM Generation](https://mlsanity.com/paper/2502.10996/) (2025) - [EWEK-QA: Enhanced Web and Efficient Knowledge Graph Retrieval for Citation-based Question Answering Systems](https://mlsanity.com/paper/2406.10393/) (2024) - [SEMMA: A Semantic Aware Knowledge Graph Foundation Model](https://mlsanity.com/paper/2505.20422/) (2025) - [Spider4SPARQL: A Complex Benchmark for Evaluating Knowledge Graph Question Answering Systems](https://mlsanity.com/paper/2309.16248/) (2023) - [Konstruktor: A Strong Baseline for Simple Knowledge Graph Question Answering](https://mlsanity.com/paper/2409.15902/) (2024) - [ARUQULA -- An LLM based Text2SPARQL Approach using ReAct and Knowledge Graph Exploration Utilities](https://mlsanity.com/paper/2510.02200/) (2025) - [InteracSPARQL: An Interactive System for SPARQL Query Refinement Using Natural Language Explanations](https://mlsanity.com/paper/2511.02002/) (2025) - [Certifying Knowledge Comprehension in LLMs](https://mlsanity.com/paper/2402.15929/) (2024) - [AutoSchema: Live Schema Grounding for Agentic Text-to-Sparql over Heterogeneous Knowledge Graphs](https://mlsanity.com/paper/2608.14228/) (2026) - [MissDiag: Diagnostic Evaluation of Incomplete-Knowledge Robustness in KGQA and KG-RAG](https://mlsanity.com/paper/2608.18489/) (2026) ## Abstract (arXiv) Data-driven systems need to be evaluated to establish trust in the scientific approach and its applicability. In particular, this is true for Knowledge Graph (KG) Question Answering (QA), where complex data structures are made accessible via natural-language interfaces. Evaluating the capabilities of these systems has been a driver for the community for more than ten years while establishing different KGQA benchmark datasets. However, comparing different approaches is cumbersome. The lack of existing and curated leaderboards leads to a missing global view over the research field and could inject mistrust into the results. In particular, the latest and most-used datasets in the KGQA community, LC-QuAD and QALD, miss providing central and up-to-date points of trust. In this paper, we survey and analyze a wide range of evaluation results with significant coverage of 100 publications and 98 systems from the last decade. We provide a new central and open leaderboard for any KGQA benchmark dataset as a focal point for the community - https://kgqa.github.io/leaderboard. Our analysis highlights existing problems during the evaluation of KGQA systems. Thus, we will point to possible improvements for future evaluations.