Benchmarking Text-to-Python against Text-to-SQL: The Impact of Explicit Logic and Ambiguity
Not on the map. Why?
Abstract (arXiv)
While Text-to-SQL remains the dominant approach for database interaction, real-world analytics increasingly require the flexibility of general-purpose programming languages such as Python or Pandas to manage file-based data and complex analytical workflows. Despite this growing need, the reliability of Text-to-Python in core data retrieval remains underexplored relative to the mature SQL ecosystem. To address this gap, we introduce BIRD-Python, a benchmark designed for cross-paradigm evaluation. We systematically refined the original dataset to reduce annotation noise and align execution semantics, thereby establishing a consistent and standardized baseline for comparison. Our analysis reveals a fundamental paradigmatic divergence: whereas SQL leverages implicit DBMS behaviors through its declarative structure, Python requires explicit procedural logic, making it highly sensitive to underspecified user intent. To mitigate this challenge, we propose the Logic Completion Framework (LCF), which resolves ambiguity by incorporating latent domain knowledge into the generation process. Experimental results show that (1) performance differences primarily stem from missing domain context rather than inherent limitations in code generation, and (2) when these gaps are addressed, Text-to-Python achieves performance parity with Text-to-SQL. These findings establish Python as a viable foundation for analytical agents-provided that systems effectively ground ambiguous natural language inputs in executable logical specifications. Resources are available at https://anonymous.4open.science/r/Bird-Python-43B7/.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 21 of its 35 references that are in the map. First posted on arXiv 22 Jan 2026.
Builds on
21 of 35 references.
References · 21
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
- 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 679
- 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 667
- 2024StarCoder 2 and The Stack v2: The Next Generationcited by 484
- 2023Structured Chain-of-Thought Prompting for Code Generationcited by 136
- 2023An Empirical Study of the Non-determinism of ChatGPT in Code Generationcited by 134
- 2024DSBench: How Far Are Data Science Agents from Becoming Data Science Experts?cited by 119
- 2024The Death of Schema Linking? Text-to-SQL in the Age of Well-Reasoned Language Modelscited by 67
- 2024What's Wrong with Your Code Generated by Large Language Models? An Extensive Studycited by 52
- 2025UTBoost: Rigorous Evaluation of Coding Agents on SWE-Benchcited by 44
- 2025DABstep: Data Agent Benchmark for Multi-step Reasoningcited by 44
9 more
- 2023Evaluating Cross-Domain Text-to-SQL Models and Benchmarkscited by 32
- 2025Uncertainty-Guided Chain-of-Thought for Code Generation with LLMscited by 30
- 2024RuleArena: A Benchmark for Rule-Guided Reasoning with LLMs in Real-World Scenarioscited by 26
- 2025BIRD-INTERACT: Re-imagining Text-to-SQL Evaluation for Large Language Models via Lens of Dynamic Interactionscited by 17
- 2024StrucText-Eval: Evaluating Large Language Model's Reasoning Ability in Structure-Rich Textcited by 13
- 2024FLEX: Expert-level False-Less EXecution Metric for Reliable Text-to-SQL Benchmarkcited by 11
- 2023Reboost Large Language Model-based Text-to-SQL, Text-to-Python, and Text-to-Function -- with Real Applications in Traffic Domaincited by 9
- 2025SQL-of-Thought: Multi-agentic Text-to-SQL with Guided Error Correctioncited by 5
- 2025How to Enable Effective Cooperation Between Humans and NLP Models: A Survey of Principles, Formalizations, and Beyondcited by 4
Built on by
Citing papers per year: 2026: 1.
1 paper.