# Auto-Cypher: Improving LLMs on Cypher generation via LLM-supervised generation-verification framework

arXiv: https://arxiv.org/abs/2412.12612 · HTML: https://mlsanity.com/paper/2412.12612/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 46 references that are in the map.
- First posted on arXiv 17 Dec 2024.

## Builds on (most cited first, 15 of 15)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Qwen2.5-Coder Technical Report](https://mlsanity.com/paper/2409.12186/) (2024)
- [Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task](https://mlsanity.com/paper/1809.08887/) (2018)
- [Enhancing Chat Language Models by Scaling High-quality Instructional Conversations](https://mlsanity.com/paper/2305.14233/) (2023)
- [WizardCoder: Empowering Code Large Language Models with Evol-Instruct](https://mlsanity.com/paper/2306.08568/) (2023)
- [Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs](https://mlsanity.com/paper/2305.03111/) (2023)
- [Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning](https://mlsanity.com/paper/1709.00103/) (2017)
- [A Survey on Knowledge Graphs: Representation, Acquisition and Applications](https://mlsanity.com/paper/2002.00388/) (2020)
- [Structure-Grounded Pretraining for Text-to-SQL](https://mlsanity.com/paper/2010.12773/) (2020)
- [A Survey on Employing Large Language Models for Text-to-SQL Tasks](https://mlsanity.com/paper/2407.15186/) (2024)
- [Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows?](https://mlsanity.com/paper/2407.10956/) (2024)
- [Dr.Spider: A Diagnostic Evaluation Benchmark towards Text-to-SQL Robustness](https://mlsanity.com/paper/2301.08881/) (2023)
- [Recent Advances in Text-to-SQL: A Survey of What We Have and What We Expect](https://mlsanity.com/paper/2208.10099/) (2022)

## Built on by (most cited first, 10 of 10)

- [M3TQA: Massively Multilingual Multitask Table Question Answering](https://mlsanity.com/paper/2508.16265/) (2025)
- [Apriel-1.5-15b-Thinker](https://mlsanity.com/paper/2510.01141/) (2025)
- [ZOGRASCOPE: A New Benchmark for Semantic Parsing over Property Graphs](https://mlsanity.com/paper/2503.05268/) (2025)
- [A Human-in-the-Loop, LLM-Centered Architecture for Knowledge-Graph Question Answering](https://mlsanity.com/paper/2602.05512/) (2026)
- [PIPE-Cypher: Automatic Enterprise Benchmark Generation for Text-to-Cypher Systems](https://mlsanity.com/paper/2606.08481/) (2026)
- [Achieving Precise Text-To-Cypher Via Grounded Knowledge Graph Data Generation](https://mlsanity.com/paper/2606.14325/) (2026)
- [Skeletons Matter: Dynamic Data Augmentation for Text-to-Query](https://mlsanity.com/paper/2511.18934/) (2025)
- [KG2Cypher: Data-Centric Pipeline for Building Enterprise Text-to-Cypher Systems](https://mlsanity.com/paper/2606.27742/) (2026)
- [What Drives Recovery in Agentic Text-to-Cypher? LAST-CQ: An LLM Agent Self-Refinement Framework](https://mlsanity.com/paper/2609.12746/) (2026)
- [CypherTurn: A Multi-Turn Benchmark for Conversational Text-to-Cypher Evaluation and the Autonomy Divergence](https://mlsanity.com/paper/2609.36987/) (2026)

## Abstract (arXiv)

Graph databases like Neo4j are gaining popularity for handling complex, interconnected data, over traditional relational databases in modeling and querying relationships. While translating natural language into SQL queries is well-researched, generating Cypher queries for Neo4j remains relatively underexplored. In this work, we present an automated, LLM-Supervised, pipeline to generate high-quality synthetic data for Text2Cypher. Our Cypher data generation pipeline introduces LLM-As-Database-Filler, a novel strategy for ensuring Cypher query correctness, thus resulting in high quality generations. Using our pipeline, we generate high quality Text2Cypher data - SynthCypher containing 29.8k instances across various domains and queries with varying complexities. Training open-source LLMs like LLaMa-3.1-8B, Mistral-7B, and QWEN-7B on SynthCypher results in performance gains of up to 40% on the Text2Cypher test split and 30% on the SPIDER benchmark, adapted for graph databases.