Ground-Truth Subgraphs for Better Training and Evaluation of Knowledge Graph Augmented LLMs
Not on the map. Why?
Abstract (arXiv)
Retrieval of information from graph-structured knowledge bases represents a promising direction for improving the factuality of LLMs. While various solutions have been proposed, a comparison of methods is difficult due to the lack of challenging QA datasets with ground-truth targets for graph retrieval. We present SynthKGQA, an LLM-powered framework for generating high-quality Knowledge Graph Question Answering datasets from any Knowledge Graph, providing the full set of ground-truth facts in the KG to reason over questions. We show how, in addition to enabling more informative benchmarking of KG retrievers, the data produced with SynthKGQA also allows us to train better models.We apply SynthKGQA to Wikidata to generate GTSQA, a new dataset designed to test zero-shot generalization abilities of KG retrievers with respect to unseen graph structures and relation types, and benchmark popular solutions for KG-augmented LLMs on it.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 30 of its 49 references that are in the map. First posted on arXiv 6 Nov 2025.
Builds on
30 of 49 references.
References · 30
- 2023GPT-4 Technical Reportcited by 18,132
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
- 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,148
- 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
- 2021Improving language models by retrieving from trillions of tokenscited by 1,278
- 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
- 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
- 2018The Web as a Knowledge-base for Answering Complex Questionscited by 409
- 2023Reasoning on Graphs: Faithful and Interpretable Large Language Model Reasoningcited by 326
18 more
- 2024Graph Retrieval-Augmented Generation: A Surveycited by 320
- 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 240
- 2020Beyond I.I.D.: Three Levels of Generalization for Question Answering on Knowledge Basescited by 169
- 2020Labeling Trick: A Theory of Using Graph Neural Networks for Multi-Node Representation Learningcited by 148
- 2021Improving Multi-hop Knowledge Base Question Answering by Learning Intermediate Supervision Signalscited by 100
- 2022Subgraph Retrieval Enhanced Model for Multi-hop Knowledge Base Question Answeringcited by 93
- 2024Simple Is Effective: The Roles of Graphs and Large Language Models in Knowledge-Graph-Based Retrieval-Augmented Generationcited by 78
- 2024Plan-on-Graph: Self-Correcting Adaptive Planning of Large Language Model on Knowledge Graphscited by 77
- 2022Mintaka: A Complex, Natural, and Multilingual Dataset for End-to-End Question Answeringcited by 71
- 2024Graph-constrained Reasoning: Faithful Reasoning on Knowledge Graphs with Large Language Modelscited by 47
- 2020Neural Entity Linking: A Survey of Models Based on Deep Learningcited by 46
- 2023YAGO 4.5: A Large and Clean Knowledge Base with a Rich Taxonomycited by 37
- 2022Knowledge Base Question Answering by Case-based Reasoning over Subgraphscited by 31
- 2024Recent Advances in Named Entity Recognition: A Comprehensive Survey and Comparative Studycited by 19
- 2023zrLLM: Zero-Shot Relational Learning on Temporal Knowledge Graphs with Large Language Modelscited by 13
- 2025Diagnosing and Addressing Pitfalls in KG-RAG Datasets: Toward More Reliable Benchmarkingcited by 11
- 2022Semantic Structure based Query Graph Prediction for Question Answering over Knowledge Graphcited by 9
- 2025Dynamic-KGQA: A Scalable Framework for Generating Adaptive Question Answering Datasetscited by 5
Built on by
Citing papers per year: 2026: 2.
2 papers.