BIRDTurk: Adaptation of the BIRD Text-to-SQL Dataset to Turkish
Not on the map. Why?
Abstract (arXiv)
Text-to-SQL systems have achieved strong performance on English benchmarks, yet their behavior in morphologically rich, low-resource languages remains largely unexplored. We introduce BIRDTurk, the first Turkish adaptation of the BIRD benchmark, constructed through a controlled translation pipeline that adapts schema identifiers to Turkish while strictly preserving the logical structure and execution semantics of SQL queries and databases. Translation quality is validated on a sample size determined by the Central Limit Theorem to ensure 95% confidence, achieving 98.15% accuracy on human-evaluated samples. Using BIRDTurk, we evaluate inference-based prompting, agentic multi-stage reasoning, and supervised fine-tuning. Our results reveal that Turkish introduces consistent performance degradation, driven by both structural linguistic divergence and underrepresentation in LLM pretraining, while agentic reasoning demonstrates stronger cross-lingual robustness. Supervised fine-tuning remains challenging for standard multilingual baselines but scales effectively with modern instruction-tuned models. BIRDTurk provides a controlled testbed for cross-lingual Text-to-SQL evaluation under realistic database conditions. We release the training and development splits to support future research.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 15 of its 30 references that are in the map. First posted on arXiv 3 Feb 2026.
Builds on
15 of 30 references.
References · 15
- 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
- 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
- 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 679
- 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 667
- 2023DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correctioncited by 465
- 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 313
- 2024Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflowscited by 223
- 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 186
- 2018Improving Text-to-SQL Evaluation Methodologycited by 133
- 2022Impact of Tokenization on Language Models: An Analysis for Turkishcited by 62
- 2020A Pilot Study of Text-to-SQL Semantic Parsing for Vietnamesecited by 36
- 2019A Pilot Study for Chinese SQL Semantic Parsingcited by 33
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Multilingual Text-to-SQL: Benchmarking the Limits of Language Models with Collaborative Language Agents
- 2025Arctic-Text2SQL-R1: Simple Rewards, Strong Reasoning in Text-to-SQL
- 2025SLM-SQL: An Exploration of Small Language Models for Text-to-SQL
- 2024Bridging the Bosphorus: Advancing Turkish Large Language Models through Strategies for Low-Resource Language Adaptation and Benchmarking
- 2025JudgeSQL: Reasoning over SQL Candidates with Weighted Consensus Tournament
- 2025SQLForge: Synthesizing Reliable and Diverse Data to Enhance Text-to-SQL Reasoning in LLMs
- 2025SQL-o1: A Self-Reward Heuristic Dynamic Search Method for Text-to-SQL
- 2025BIRD-INTERACT: Re-imagining Text-to-SQL Evaluation for Large Language Models via Lens of Dynamic Interactions