# Text2Cypher Across Languages: Evaluating and Finetuning LLMs

arXiv: https://arxiv.org/abs/2506.21445 · HTML: https://mlsanity.com/paper/2506.21445/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 42 references that are in the map.
- First posted on arXiv 26 Jun 2025.

## Builds on (most cited first, 30 of 32)

- [Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task](https://mlsanity.com/paper/1809.08887/) (2018)
- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [Knowledge Graphs](https://mlsanity.com/paper/2003.02320/) (2020)
- [Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model](https://mlsanity.com/paper/2402.07827/) (2024)
- [CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task](https://mlsanity.com/paper/2209.06243/) (2022)
- [Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.16039/) (2023)
- [Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Prompting](https://mlsanity.com/paper/2305.07004/) (2023)
- [LLaMA Beyond English: An Empirical Study on Language Capability Transfer](https://mlsanity.com/paper/2401.01055/) (2024)
- [Do Multilingual Language Models Think Better in English?](https://mlsanity.com/paper/2308.01223/) (2023)
- [Bactrian-X: Multilingual Replicable Instruction-Following Models with Low-Rank Adaptation](https://mlsanity.com/paper/2305.15011/) (2023)
- [Do Multilingual LLMs Think In English?](https://mlsanity.com/paper/2502.15603/) (2025)
- [Extrapolating Large Language Models to Non-English by Aligning Languages](https://mlsanity.com/paper/2308.04948/) (2023)
- [BigTranslate: Augmenting Large Language Models with Multilingual Translation Capability over 100 Languages](https://mlsanity.com/paper/2305.18098/) (2023)
- [LLMs Beyond English: Scaling the Multilingual Capability of LLMs with Cross-Lingual Feedback](https://mlsanity.com/paper/2406.01771/) (2024)
- [Beyond English-Centric LLMs: What Language Do Multilingual Language Models Think in?](https://mlsanity.com/paper/2408.10811/) (2024)
- [Lost in Translation: Large Language Models in Non-English Content Analysis](https://mlsanity.com/paper/2306.07377/) (2023)
- [A Pilot Study for Chinese SQL Semantic Parsing](https://mlsanity.com/paper/1909.13293/) (2019)
- [Text2Cypher: Bridging Natural Language and Graph Databases](https://mlsanity.com/paper/2412.10064/) (2024)
- [MEXA: Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignment](https://mlsanity.com/paper/2410.05873/) (2024)
- [Role of Language Relatedness in Multilingual Fine-tuning of Language Models: A Case Study in Indo-Aryan Languages](https://mlsanity.com/paper/2109.10534/) (2021)
- [MultiSpider: Towards Benchmarking Multilingual Text-to-SQL Semantic Parsing](https://mlsanity.com/paper/2212.13492/) (2022)
- [Layer Swapping for Zero-Shot Cross-Lingual Transfer in Large Language Models](https://mlsanity.com/paper/2410.01335/) (2024)
- [mRAT-SQL+GAP:A Portuguese Text-to-SQL Transformer](https://mlsanity.com/paper/2110.03546/) (2021)
- [Setting Standards in Turkish NLP: TR-MMLU for Large Language Model Evaluation](https://mlsanity.com/paper/2501.00593/) (2024)
- [Compositional Generalization in Multilingual Semantic Parsing over Wikidata](https://mlsanity.com/paper/2108.03509/) (2021)
- [StatBot.Swiss: Bilingual Open Data Exploration in Natural Language](https://mlsanity.com/paper/2406.03170/) (2024)
- [Ar-Spider: Text-to-SQL in Arabic](https://mlsanity.com/paper/2402.15012/) (2024)
- [Why Not Transform Chat Large Language Models to Non-English?](https://mlsanity.com/paper/2405.13923/) (2024)
- [MST5 -- Multilingual Question Answering over Knowledge Graphs](https://mlsanity.com/paper/2407.06041/) (2024)
- [Multilingual Prompts in LLM-Based Recommenders: Performance Across Languages](https://mlsanity.com/paper/2409.07604/) (2024)

## Built on by (most cited first, 3 of 3)

- [CYGNET: Cypher Gate for Neural Execution Triage and Cost Containment](https://mlsanity.com/paper/2606.04645/) (2026)
- [Incremental Multilingual Text2Cypher with Adapter Combination](https://mlsanity.com/paper/2601.16097/) (2026)
- [KG2Cypher: Data-Centric Pipeline for Building Enterprise Text-to-Cypher Systems](https://mlsanity.com/paper/2606.27742/) (2026)

## Abstract (arXiv)

Recent advances in large language models (LLMs) have enabled natural language interfaces that translate user questions into database queries, such as Text2SQL, Text2SPARQL, and Text2Cypher. While these interfaces enhance database accessibility, most research today focuses on English, with limited evaluation in other languages. This paper investigates the performance of both foundational and finetuned LLMs on the Text2Cypher task across multiple languages. We create and release a multilingual dataset by translating English questions into Spanish and Turkish while preserving the original Cypher queries, enabling fair cross-lingual comparison. Using standardized prompts and metrics, we evaluate several foundational models and observe a consistent performance pattern: highest on English, followed by Spanish, and lowest on Turkish. We attribute this to differences in training data availability and linguistic features. We also examine the impact of translating task prompts into Spanish and Turkish. Results show little to no change in evaluation metrics, suggesting prompt translation has minor impact. Furthermore, we finetune a foundational model on two datasets: one in English only, and one multilingual. Finetuning on English improves overall accuracy but widens the performance gap between languages. In contrast, multilingual finetuning narrows the gap, resulting in more balanced performance. Our findings highlight the importance for multilingual evaluation and training to build more inclusive and robust query generation systems.