mlsanityData to 14 Oct 2026

BIRD-INTERACT: Re-imagining Text-to-SQL Evaluation for Large Language Models via Lens of Dynamic Interactions

arXiv 2510.05318 ↗First posted 6 Oct 2025cs.AIcited by 19

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have demonstrated remarkable performance on single-turn text-to-SQL tasks, but real-world database applications predominantly require multi-turn interactions to handle ambiguous queries, execution errors, and evolving user requirements. Existing multi-turn benchmarks fall short by treating conversation histories as static context or limiting evaluation to read-only operations, failing to reflect production-grade database assistant challenges. We introduce BIRD-INTERACT, a benchmark that restores this realism through: (1) a comprehensive interaction environment coupling each database with a hierarchical knowledge base, metadata files, and a function-driven user simulator, enabling models to solicit clarifications, retrieve knowledge, and recover from errors without human supervision; (2) two evaluation settings consisting of a pre-defined conversational protocol (c-Interact) and an open-ended agentic setting (a-Interact) where models autonomously decide when to query the user simulator or explore the environment; (3) a challenging task suite covering the full CRUD spectrum for business-intelligence and operational use cases, guarded by executable test cases. Each task features ambiguous and follow-up sub-tasks requiring dynamic interaction. The suite comprises BIRD-INTERACT-FULL (600 tasks, up to 11,796 interactions) for comprehensive performance assessment, and BIRD-INTERACT-LITE (300 tasks with simplified databases) for detailed behavioral analysis and rapid method development. Our empirical results highlight BIRD-INTERACT's difficulty: GPT-5 completes only 8.67% of tasks in c-Interact and 17.00% in a-Interact. Analysis via memory grafting and Interaction Test-time Scaling validates the importance of effective interaction for complex, dynamic text-to-SQL tasks.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Builds on 65 of its 92 references that are in the map. First posted on arXiv 6 Oct 2025.

Builds on

65 of 92 references.

References · 65

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  4. 2024A Survey on LLM-as-a-Judgecited by 1,423
  5. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  6. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  7. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,139
  8. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 688
  9. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 687
  10. 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 627
  11. 2018Wizard of Wikipedia: Knowledge-Powered Conversational agentscited by 530
  12. 2023DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correctioncited by 477
38 more
  1. 2019RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parserscited by 424
  2. 2020AmbigQA: Answering Ambiguous Open-domain Questionscited by 390
  3. 2023Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluationcited by 385
  4. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 297
  5. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  6. 2024FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AIcited by 243
  7. 2023InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedbackcited by 238
  8. 2024Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflowscited by 232
  9. 2024CodeS: Towards Building Open-source Language Models for Text-to-SQLcited by 187
  10. 2019CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databasescited by 185
  11. 2024CHESS: Contextual Harnessing for Efficient SQL Synthesiscited by 169
  12. 2023MAC-SQL: A Multi-Agent Collaborative Framework for Text-to-SQLcited by 164
  13. 2019SParC: Cross-Domain Semantic Parsing in Contextcited by 144
  14. 2024CHASE-SQL: Multi-Path Reasoning and Preference Optimized Candidate Selection in Text-to-SQLcited by 138
  15. 2025CollabLLM: From Passive Responders to Active Collaboratorscited by 124
  16. 2025SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Taskscited by 118
  17. 2021KaggleDBQA: Realistic Evaluation of Text-to-SQL Parserscited by 106
  18. 2024MCS-SQL: Leveraging Multiple Prompts and Multiple-Choice Selection For Text-to-SQL Generationcited by 105
  19. 2023Graphix-T5: Mixing Pre-Trained Transformers with Graph-Aware Layers for Text-to-SQL Parsingcited by 93
  20. 2025OmniSQL: Synthesizing High-quality Text-to-SQL Data at Scalecited by 91
  21. 2022Natural Language to Code Generation in Interactive Data Science Notebookscited by 87
  22. 2024A Survey on Employing Large Language Models for Text-to-SQL Taskscited by 67
  23. 2024The Death of Schema Linking? Text-to-SQL in the Age of Well-Reasoned Language Modelscited by 67
  24. 2025Reasoning-SQL: Reinforcement Learning with SQL Tailored Partial Rewards for Reasoning-Enhanced Text-to-SQLcited by 59
  25. 2024Learning to Clarify: Multi-turn Conversations with Action-Based Contrastive Self-Trainingcited by 57
  26. 2024RSL-SQL: Robust Schema Linking in Text-to-SQL Generationcited by 52
  27. 2024Before Generation, Align it! A Novel and Effective Strategy for Mitigating Hallucinations in Text-to-SQL Generationcited by 50
  28. 2024DTS-SQL: Decomposed Text-to-SQL with Small Large Language Modelscited by 45
  29. 2021Text-to-SQL in the Wild: A Naturally-Occurring Dataset Based on Stack Exchange Datacited by 43
  30. 2025XiYan-SQL: A Novel Multi-Generator Framework For Text-to-SQLcited by 40
  31. 2024A Taxonomy for Human-LLM Interaction Modes: An Initial Explorationcited by 38
  32. 2023Re-Reading Improves Reasoning in Large Language Modelscited by 37
  33. 2024AMBROSIA: A Benchmark for Parsing Ambiguous Questions into Database Queriescited by 35
  34. 2024E-SQL: Direct Schema Linking via Question Enrichment in Text-to-SQLcited by 35
  35. 2023Benchmarking and Improving Text-to-SQL Generation under Ambiguitycited by 34
  36. 2025Alpha-SQL: Zero-Shot Text-to-SQL using Monte Carlo Tree Searchcited by 34
  37. 2025Steering LLM Thinking with Budget Guidancecited by 33
  38. 2023PlatoLM: Teaching LLMs in Multi-Round Dialogue via a User Simulatorcited by 29

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 15.

19 papers.

Sort:

By area:

  1. 2025DAComp: Benchmarking Data Agents across the Full Data Intelligence Lifecyclecited by 23
  2. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluationcited by 7
  3. 2025Agent Bain vs. Agent McKinsey: A New Text-to-SQL Benchmark for the Business Domaincited by 4
  4. 2025Reasoning about Intent for Ambiguous Requestscited by 4
  5. 2026Learning to Retrieve: Dual-Level Long-Term Memory for Text-to-SQL Agentscited by 4
  6. 2026SQL-ASTRA: Alleviating Sparse Feedback in Agentic SQL via Column-Set Matching and Trajectory Aggregationcited by 3
  7. 2026PrepBench: How Far Are We from Natural-Language-Driven Data Preparation?cited by 3
  8. 2026CLARITY: A Framework and Benchmark for Conversational Language Ambiguity and Unanswerability in Interactive NL2SQL Systemscited by 2
  9. 2026GDPevo: Evaluating Agent Self-Evolution on Real Business Taskscited by 2
  10. 2026Benchmarking Text-to-Python against Text-to-SQL: The Impact of Explicit Logic and Ambiguitycited by 1
  11. 2026DV-World: Benchmarking Data Visualization Agents in Real-World Scenarioscited by 1
  12. 2026BAP-SQL: Budget-Aware Observation Planning for Agentic Text-to-SQLcited by 1
7 more
  1. 2026Beyond Oracle Communication: Benchmarking Interactive Intent Alignment Under Miscommunication and Evolving User Intentcited by 1
  2. 2025Can AI autonomously build, operate, and use the entire data stack?cited by 0
  3. 2026Memory Architectures for Multi-Turn Text-to-SQL: A Benchmark and Empirical Studycited by 0
  4. 2026Data Intelligence Agents: Interpreting, Modeling, and Querying Enterprise Data via Autonomous Coding Agentscited by 0
  5. 2026PLSQLBench: Benchmarking LLM Systems for Executable Procedural Database Programmingcited by 0
  6. 2026ARCS: Towards Precise Text-to-SQL via Structured Disambiguationcited by 0
  7. 2026HarnessSQL: Harness-Native Training for SQL Agents in Realistic Database Environmentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Rethinking Text-to-SQL: Dynamic Multi-turn SQL Interaction for Real-world Database Exploration
  2. 2024Multi-Turn Interactions for Text-to-SQL with Large Language Models
  3. 2025BAPPA: Benchmarking Agents, Plans, and Pipelines for Automated Text-to-SQL Generation
  4. 2025RAISE: Reasoning Agent for Interactive SQL Exploration
  5. 2025MTSQL-R1: Towards Long-Horizon Multi-Turn Text-to-SQL via Agentic Training
  6. 2025AskDB: An LLM Agent for Natural Language Interaction with Relational Databases
  7. 2023On Evaluating the Integration of Reasoning and Action in LLM Agents with Database Question Answering
  8. 2025Multilingual Text-to-SQL: Benchmarking the Limits of Language Models with Collaborative Language Agents