# ZOGRASCOPE: A New Benchmark for Semantic Parsing over Property Graphs

arXiv: https://arxiv.org/abs/2503.05268 · HTML: https://mlsanity.com/paper/2503.05268/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 22 references that are in the map.
- First posted on arXiv 7 Mar 2025.

## Builds on (most cited first, 14 of 14)

- [Variational Reasoning for Question Answering with Knowledge Graph](https://mlsanity.com/paper/1709.04071/) (2017)
- [Few-shot In-context Learning for Knowledge Base Question Answering](https://mlsanity.com/paper/2305.01750/) (2023)
- [ArcaneQA: Dynamic Program Induction and Contextualized Encoding for Knowledge Base Question Answering](https://mlsanity.com/paper/2204.08109/) (2022)
- [Text2Cypher: Bridging Natural Language and Graph Databases](https://mlsanity.com/paper/2412.10064/) (2024)
- [SLOG: A Structural Generalization Benchmark for Semantic Parsing](https://mlsanity.com/paper/2310.15040/) (2023)
- [SyntheT2C: Generating Synthetic Data for Fine-Tuning Large Language Models on the Text2Cypher Task](https://mlsanity.com/paper/2406.10710/) (2024)
- [CypherBench: Towards Precise Retrieval over Full-scale Modern Knowledge Graphs in the LLM Era](https://mlsanity.com/paper/2412.18702/) (2024)
- [Aligning Large Language Models to a Domain-specific Graph Database for NL2GQL](https://mlsanity.com/paper/2402.16567/) (2024)
- [GraphQ IR: Unifying the Semantic Parsing of Graph Query Languages with One Intermediate Representation](https://mlsanity.com/paper/2205.12078/) (2022)
- [Bring Your Own KG: Self-Supervised Program Synthesis for Zero-Shot KGQA](https://mlsanity.com/paper/2311.07850/) (2023)
- [The Gremlin Graph Traversal Machine and Language](https://mlsanity.com/paper/1508.03843/) (2015)
- [Auto-Cypher: Improving LLMs on Cypher generation via LLM-supervised generation-verification framework](https://mlsanity.com/paper/2412.12612/) (2024)
- [AMR Parsing is Far from Solved: GrAPES, the Granular AMR Parsing Evaluation Suite](https://mlsanity.com/paper/2312.03480/) (2023)
- [Rel2Graph: Automated Mapping From Relational Databases to a Unified Property Knowledge Graph](https://mlsanity.com/paper/2310.01080/) (2023)

## Built on by (most cited first, 1 of 1)

- [CypherTurn: A Multi-Turn Benchmark for Conversational Text-to-Cypher Evaluation and the Autonomy Divergence](https://mlsanity.com/paper/2609.36987/) (2026)

## Abstract (arXiv)

In recent years, the need for natural language interfaces to knowledge graphs has become increasingly important since they enable easy and efficient access to the information contained in them. In particular, property graphs (PGs) have seen increased adoption as a means of representing complex structured information. Despite their growing popularity in industry, PGs remain relatively underrepresented in semantic parsing research with a lack of resources for evaluation. To address this gap, we introduce ZOGRASCOPE, a benchmark designed specifically for PGs and queries written in Cypher. Our benchmark includes a diverse set of manually annotated queries of varying complexity and is organized into three partitions: iid, compositional and length. We complement this paper with a set of experiments that test the performance of different LLMs in a variety of learning settings.