# CypherBench: Towards Precise Retrieval over Full-scale Modern Knowledge Graphs in the LLM Era

arXiv: https://arxiv.org/abs/2412.18702 · HTML: https://mlsanity.com/paper/2412.18702/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 77 references that are in the map.
- First posted on arXiv 24 Dec 2024.

## Builds on (most cited first, 30 of 42)

- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [From Local to Global: A Graph RAG Approach to Query-Focused Summarization](https://mlsanity.com/paper/2404.16130/) (2024)
- [Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task](https://mlsanity.com/paper/1809.08887/) (2018)
- [Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs](https://mlsanity.com/paper/2305.03111/) (2023)
- [The Web as a Knowledge-base for Answering Complex Questions](https://mlsanity.com/paper/1803.06643/) (2018)
- [QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering](https://mlsanity.com/paper/2104.06378/) (2021)
- [Reasoning on Graphs: Faithful and Interpretable Large Language Model Reasoning](https://mlsanity.com/paper/2310.01061/) (2023)
- [Graph Retrieval-Augmented Generation: A Survey](https://mlsanity.com/paper/2408.08921/) (2024)
- [Beyond I.I.D.: Three Levels of Generalization for Question Answering on Knowledge Bases](https://mlsanity.com/paper/2011.07743/) (2020)
- [Variational Reasoning for Question Answering with Knowledge Graph](https://mlsanity.com/paper/1709.04071/) (2017)
- [Scalable Multi-Hop Relational Reasoning for Knowledge-Aware Question Answering](https://mlsanity.com/paper/2005.00646/) (2020)
- [UniKGQA: Unified Retrieval and Reasoning for Solving Multi-hop Question Answering Over Knowledge Graph](https://mlsanity.com/paper/2212.00959/) (2022)
- [Compositional Generalization in Semantic Parsing: Pre-training vs. Specialized Architectures](https://mlsanity.com/paper/2007.08970/) (2020)
- [Subgraph Retrieval Enhanced Model for Multi-hop Knowledge Base Question Answering](https://mlsanity.com/paper/2202.13296/) (2022)
- [DecAF: Joint Decoding of Answers and Logical Forms for Question Answering over Knowledge Bases](https://mlsanity.com/paper/2210.00063/) (2022)
- [KQA Pro: A Dataset with Explicit Compositional Programs for Complex Question Answering over Knowledge Base](https://mlsanity.com/paper/2007.03875/) (2020)
- [Few-shot In-context Learning for Knowledge Base Question Answering](https://mlsanity.com/paper/2305.01750/) (2023)
- [UniK-QA: Unified Representations of Structured and Unstructured Knowledge for Open-Domain Question Answering](https://mlsanity.com/paper/2012.14610/) (2020)
- [FactKG: Fact Verification via Reasoning on Knowledge Graphs](https://mlsanity.com/paper/2305.06590/) (2023)
- [STaRK: Benchmarking LLM Retrieval on Textual and Relational Knowledge Bases](https://mlsanity.com/paper/2404.13207/) (2024)
- [Paths-over-Graph: Knowledge Graph Empowered Large Language Model Reasoning](https://mlsanity.com/paper/2410.14211/) (2024)
- [Multi-hop Question Answering under Temporal Knowledge Editing](https://mlsanity.com/paper/2404.00492/) (2024)
- [Modern Baselines for SPARQL Semantic Parsing](https://mlsanity.com/paper/2204.12793/) (2022)
- [Open Domain Question Answering with A Unified Knowledge Interface](https://mlsanity.com/paper/2110.08417/) (2021)
- [SPINACH: SPARQL-Based Information Navigation for Challenging Real-World Questions](https://mlsanity.com/paper/2407.11417/) (2024)
- [Conversational Question Answering on Heterogeneous Sources](https://mlsanity.com/paper/2204.11677/) (2022)
- [Direct Fact Retrieval from Knowledge Graphs without Entity Linking](https://mlsanity.com/paper/2305.12416/) (2023)
- [LLM-based SPARQL Query Generation from Natural Language over Federated Knowledge Graphs](https://mlsanity.com/paper/2410.06062/) (2024)
- [Falcon 2.0: An Entity and Relation Linking Tool over Wikidata](https://mlsanity.com/paper/1912.11270/) (2019)
- [SyntheT2C: Generating Synthetic Data for Fine-Tuning Large Language Models on the Text2Cypher Task](https://mlsanity.com/paper/2406.10710/) (2024)

## Built on by (most cited first, 16 of 16)

- [The Great Nugget Recall: Automating Fact Extraction and RAG Evaluation with Large Language Models](https://mlsanity.com/paper/2504.15068/) (2025)
- [Multi-Agent GraphRAG: A Text-to-Cypher Framework for Labeled Property Graphs](https://mlsanity.com/paper/2511.08274/) (2025)
- [Text2GraphQuery-Bench: A Text to Graph Query Benchmark](https://mlsanity.com/paper/2602.11745/) (2026)
- [KAQG: A Knowledge-Graph-Enhanced RAG for Difficulty-Controlled Question Generation](https://mlsanity.com/paper/2505.07618/) (2025)
- [ZOGRASCOPE: A New Benchmark for Semantic Parsing over Property Graphs](https://mlsanity.com/paper/2503.05268/) (2025)
- [A Human-in-the-Loop, LLM-Centered Architecture for Knowledge-Graph Question Answering](https://mlsanity.com/paper/2602.05512/) (2026)
- [GraphWalk: Enabling Reasoning in Large Language Models through Tool-Based Graph Navigation](https://mlsanity.com/paper/2604.01610/) (2026)
- [CYGNET: Cypher Gate for Neural Execution Triage and Cost Containment](https://mlsanity.com/paper/2606.04645/) (2026)
- [PIPE-Cypher: Automatic Enterprise Benchmark Generation for Text-to-Cypher Systems](https://mlsanity.com/paper/2606.08481/) (2026)
- [Achieving Precise Text-To-Cypher Via Grounded Knowledge Graph Data Generation](https://mlsanity.com/paper/2606.14325/) (2026)
- [Graph Query Generation with Constraint-guided Large Language Agents](https://mlsanity.com/paper/2605.00845/) (2026)
- [Reducing Hallucinations in Complex Question Answering using Simple Graph-based Retrieval-Augmented Generation (long version)](https://mlsanity.com/paper/2606.05901/) (2026)
- [CypherTurn: A Multi-Turn Benchmark for Conversational Text-to-Cypher Evaluation and the Autonomy Divergence](https://mlsanity.com/paper/2609.36987/) (2026)
- [CultureBot: A Framework for Retrieval-Augmented Exploration of Cultural Heritage Data](https://mlsanity.com/paper/2610.16435/) (2026)
- [Knowledge Graphs as Files: An Agent Harness Where Tool Creation and Tool Use Co-Evolve](https://mlsanity.com/paper/2610.18616/) (2026)
- [HERO: Harnessing Evidence for Test-time Refinement and Prompt Optimization in Program Generation](https://mlsanity.com/paper/2610.19184/) (2026)

## Abstract (arXiv)

Retrieval from graph data is crucial for augmenting large language models (LLM) with both open-domain knowledge and private enterprise data, and it is also a key component in the recent GraphRAG system (edge et al., 2024). Despite decades of research on knowledge graphs and knowledge base question answering, leading LLM frameworks (e.g. Langchain and LlamaIndex) have only minimal support for retrieval from modern encyclopedic knowledge graphs like Wikidata. In this paper, we analyze the root cause and suggest that modern RDF knowledge graphs (e.g. Wikidata, Freebase) are less efficient for LLMs due to overly large schemas that far exceed the typical LLM context window, use of resource identifiers, overlapping relation types and lack of normalization. As a solution, we propose property graph views on top of the underlying RDF graph that can be efficiently queried by LLMs using Cypher. We instantiated this idea on Wikidata and introduced CypherBench, the first benchmark with 11 large-scale, multi-domain property graphs with 7.8 million entities and over 10,000 questions. To achieve this, we tackled several key challenges, including developing an RDF-to-property graph conversion engine, creating a systematic pipeline for text-to-Cypher task generation, and designing new evaluation metrics.