mlsanityData to 7 Oct 2026

CypherBench: Towards Precise Retrieval over Full-scale Modern Knowledge Graphs in the LLM Era

arXiv 2412.18702 ↗First posted 24 Dec 2024cs.CL cs.AI cs.DBcited by 13

Not on the map. Why?

Abstract (arXiv)

Retrieval from graph data is crucial for augmenting large language models (LLM) with both open-domain knowledge and private enterprise data, and it is also a key component in the recent GraphRAG system (edge et al., 2024). Despite decades of research on knowledge graphs and knowledge base question answering, leading LLM frameworks (e.g. Langchain and LlamaIndex) have only minimal support for retrieval from modern encyclopedic knowledge graphs like Wikidata. In this paper, we analyze the root cause and suggest that modern RDF knowledge graphs (e.g. Wikidata, Freebase) are less efficient for LLMs due to overly large schemas that far exceed the typical LLM context window, use of resource identifiers, overlapping relation types and lack of normalization. As a solution, we propose property graph views on top of the underlying RDF graph that can be efficiently queried by LLMs using Cypher. We instantiated this idea on Wikidata and introduced CypherBench, the first benchmark with 11 large-scale, multi-domain property graphs with 7.8 million entities and over 10,000 questions. To achieve this, we tackled several key challenges, including developing an RDF-to-property graph conversion engine, creating a systematic pipeline for text-to-Cypher task generation, and designing new evaluation metrics.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 42 of its 77 references that are in the map. First posted on arXiv 24 Dec 2024.

Builds on

42 of 77 references.

References · 42

  1. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  2. 2024From Local to Global: A Graph RAG Approach to Query-Focused Summarizationcited by 1,411
  3. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  4. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 667
  5. 2018The Web as a Knowledge-base for Answering Complex Questionscited by 409
  6. 2021QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answeringcited by 386
  7. 2023Reasoning on Graphs: Faithful and Interpretable Large Language Model Reasoningcited by 326
  8. 2024Graph Retrieval-Augmented Generation: A Surveycited by 320
  9. 2017Variational Reasoning for Question Answering with Knowledge Graphcited by 170
  10. 2020Beyond I.I.D.: Three Levels of Generalization for Question Answering on Knowledge Basescited by 169
  11. 2020Scalable Multi-Hop Relational Reasoning for Knowledge-Aware Question Answeringcited by 122
  12. 2022UniKGQA: Unified Retrieval and Reasoning for Solving Multi-hop Question Answering Over Knowledge Graphcited by 100
30 more
  1. 2020Compositional Generalization in Semantic Parsing: Pre-training vs. Specialized Architecturescited by 95
  2. 2022Subgraph Retrieval Enhanced Model for Multi-hop Knowledge Base Question Answeringcited by 93
  3. 2022DecAF: Joint Decoding of Answers and Logical Forms for Question Answering over Knowledge Basescited by 83
  4. 2020KQA Pro: A Dataset with Explicit Compositional Programs for Complex Question Answering over Knowledge Basecited by 76
  5. 2023Few-shot In-context Learning for Knowledge Base Question Answeringcited by 74
  6. 2020UniK-QA: Unified Representations of Structured and Unstructured Knowledge for Open-Domain Question Answeringcited by 61
  7. 2023FactKG: Fact Verification via Reasoning on Knowledge Graphscited by 57
  8. 2024STaRK: Benchmarking LLM Retrieval on Textual and Relational Knowledge Basescited by 53
  9. 2024Paths-over-Graph: Knowledge Graph Empowered Large Language Model Reasoningcited by 44
  10. 2024Multi-hop Question Answering under Temporal Knowledge Editingcited by 36
  11. 2022Modern Baselines for SPARQL Semantic Parsingcited by 34
  12. 2021Open Domain Question Answering with A Unified Knowledge Interfacecited by 32
  13. 2024SPINACH: SPARQL-Based Information Navigation for Challenging Real-World Questionscited by 27
  14. 2022Conversational Question Answering on Heterogeneous Sourcescited by 26
  15. 2023Direct Fact Retrieval from Knowledge Graphs without Entity Linkingcited by 22
  16. 2024LLM-based SPARQL Query Generation from Natural Language over Federated Knowledge Graphscited by 22
  17. 2019Falcon 2.0: An Entity and Relation Linking Tool over Wikidatacited by 21
  18. 2024SyntheT2C: Generating Synthetic Data for Fine-Tuning Large Language Models on the Text2Cypher Taskcited by 19
  19. 2020KGTK: A Toolkit for Large Knowledge Graph Manipulation and Analysiscited by 17
  20. 2021Formal Query Building with Query Structure Prediction for Complex Question Answering over Knowledge Basecited by 16
  21. 2024Aligning Large Language Models to a Domain-specific Graph Database for NL2GQLcited by 14
  22. 2022GraphQ IR: Unifying the Semantic Parsing of Graph Query Languages with One Intermediate Representationcited by 13
  23. 2021Beyond NED: Fast and Effective Search Space Reduction for Complex Question Answering over Knowledge Basescited by 11
  24. 2023Explainable Conversational Question Answering over Heterogeneous Sources via Iterative Graph Neural Networkscited by 9
  25. 2023R3-NL2GQL: A Model Coordination and Knowledge Graph Alignment Approach for NL2GQLcited by 7
  26. 2014Reconciliation of RDF* and Property Graphscited by 6
  27. 2023Calibrated Seq2seq Models for Efficient and Generalizable Ultra-fine Entity Typingcited by 5
  28. 2024UniOQA: A Unified Framework for Knowledge Graph Question Answering with Large Language Modelscited by 5
  29. 2024SPARKLE: Enhancing SPARQL Generation with Direct KG Integration in Decodingcited by 5
  30. 2018Mapping RDF Graphs to Property Graphscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 9.

13 papers.

Sort:

By area:

  1. 2025The Great Nugget Recall: Automating Fact Extraction and RAG Evaluation with Large Language Modelscited by 24
  2. 2025Multi-Agent GraphRAG: A Text-to-Cypher Framework for Labeled Property Graphscited by 5
  3. 2026Text2GraphQuery-Bench: A Text to Graph Query Benchmarkcited by 3
  4. 2025KAQG: A Knowledge-Graph-Enhanced RAG for Difficulty-Controlled Question Generationcited by 2
  5. 2025ZOGRASCOPE: A New Benchmark for Semantic Parsing over Property Graphscited by 1
  6. 2026A Human-in-the-Loop, LLM-Centered Architecture for Knowledge-Graph Question Answeringcited by 1
  7. 2026GraphWalk: Enabling Reasoning in Large Language Models through Tool-Based Graph Navigationcited by 1
  8. 2026CYGNET: Cypher Gate for Neural Execution Triage and Cost Containmentcited by 1
  9. 2026Achieving Precise Text-To-Cypher Via Grounded Knowledge Graph Data Generationcited by 1
  10. 2026Graph Query Generation with Constraint-guided Large Language Agentscited by 0
  11. 2026Reducing Hallucinations in Complex Question Answering using Simple Graph-based Retrieval-Augmented Generation (long version)cited by 0
  12. 2026PIPE-Cypher: Automatic Enterprise Benchmark Generation for Text-to-Cypher Systemscited by 0
1 more
  1. 2026CypherTurn: A Multi-Turn Benchmark for Conversational Text-to-Cypher Evaluation and the Autonomy Divergencecited by 0