mlsanityData to 14 Oct 2026

Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows

arXiv 2411.07763 ↗First posted 12 Nov 2024cs.CL cs.AI cs.DBcited by 232

Not on the map. Why?

Cited by 64 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Real-world enterprise text-to-SQL workflows often involve complex cloud or local data across various database systems, multiple SQL queries in various dialects, and diverse operations from data transformation to analytics. We introduce Spider 2.0, an evaluation framework comprising 632 real-world text-to-SQL workflow problems derived from enterprise-level database use cases. The databases in Spider 2.0 are sourced from real data applications, often containing over 1,000 columns and stored in local or cloud database systems such as BigQuery and Snowflake. We show that solving problems in Spider 2.0 frequently requires understanding and searching through database metadata, dialect documentation, and even project-level codebases. This challenge calls for models to interact with complex SQL workflow environments, process extremely long contexts, perform intricate reasoning, and generate multiple SQL queries with diverse operations, often exceeding 100 lines, which goes far beyond traditional text-to-SQL challenges. Our evaluations indicate that based on o1-preview, our code agent framework successfully solves only 21.3% of the tasks, compared with 91.2% on Spider 1.0 and 73.0% on BIRD. Our results on Spider 2.0 show that while language models have demonstrated remarkable performance in code generation -- especially in prior text-to-SQL benchmarks -- they require significant improvement in order to achieve adequate performance for real-world enterprise usage. Progress on Spider 2.0 represents crucial steps towards developing intelligent, autonomous, code agents for real-world enterprise settings. Our code, baseline models, and data are available at https://spider2-sql.github.io

Key facts. Cited by 232 papers in the map (data through 14 Oct 2026). Builds on 35 of its 48 references that are in the map. First posted on arXiv 12 Nov 2024.

Builds on

35 of 48 references.

References · 35

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  4. 2021Program Synthesis with Large Language Modelscited by 3,637
  5. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  7. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,139
  8. 2023Teaching Large Language Models to Self-Debugcited by 988
  9. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 688
  10. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 687
  11. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Modelscited by 560
  12. 2023DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correctioncited by 477
23 more
  1. 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineeringcited by 469
  2. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 452
  3. 2024Agentless: Demystifying LLM-based Software Engineering Agentscited by 421
  4. 2023Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluationcited by 385
  5. 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 331
  6. 2024AutoCodeRover: Autonomous Program Improvementcited by 278
  7. 2023InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedbackcited by 238
  8. 2024CodeS: Towards Building Open-source Language Models for Text-to-SQLcited by 187
  9. 2023Planning with Large Language Models for Code Generationcited by 182
  10. 2019SPoC: Search-based Pseudocode to Codecited by 176
  11. 2024CHESS: Contextual Harnessing for Efficient SQL Synthesiscited by 169
  12. 2023C3: Zero-shot Text-to-SQL with ChatGPTcited by 164
  13. 2024Autonomous Evaluation and Refinement of Digital Agentscited by 137
  14. 2024DSBench: How Far Are Data Science Agents from Becoming Data Science Experts?cited by 125
  15. 2021KaggleDBQA: Realistic Evaluation of Text-to-SQL Parserscited by 106
  16. 2018NL2Bash: A Corpus and Semantic Parser for Natural Language Interface to the Linux Operating Systemcited by 98
  17. 2024CodeR: Issue Resolving with Multi-Agent and Task Graphscited by 91
  18. 2022Natural Language to Code Generation in Interactive Data Science Notebookscited by 87
  19. 2019Text-to-SQL Generation for Question Answering on Electronic Medical Recordscited by 77
  20. 2023ACT-SQL: In-Context Learning for Text-to-SQL with Automatically-Generated Chain-of-Thoughtcited by 65
  21. 2024DA-Code: Agent Data Science Code Generation Benchmark for Large Language Modelscited by 58
  22. 2024Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows?cited by 56
  23. 2021Text-to-SQL in the Wild: A Naturally-Occurring Dataset Based on Stack Exchange Datacited by 43

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 87; 2026: 143.

232 papers.

Sort:

By area:

  1. 2025LLMs Get Lost In Multi-Turn Conversationcited by 429
  2. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 320
  3. 2025MLGym: A New Framework and Benchmark for Advancing AI Research Agentscited by 124
  4. 2025xbench: Tracking Agents Productivity Scaling with Profession-Aligned Real-World Evaluationscited by 120
  5. 2025WideSearch: Benchmarking Agentic Broad Info-Seekingcited by 69
  6. 2026AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applicationscited by 53
  7. 2025Griffin: Towards a Graph-Centric Relational Database Foundation Modelcited by 47
  8. 2025DABstep: Data Agent Benchmark for Multi-step Reasoningcited by 46
  9. 2025ReFoRCE: A Text-to-SQL Agent with Self-Refinement, Consensus Enforcement, and Column Explorationcited by 37
  10. 2025KramaBench: A Benchmark for AI Systems on Data-to-Insight Pipelines over Data Lakescited by 33
  11. 2025LinkAlign: Scalable Schema Linking for Real-World Large-Scale Multi-Database Text-to-SQLcited by 29
  12. 2025AI Agentic Programming: A Survey of Techniques, Challenges, and Opportunitiescited by 28
38 more
  1. 2025Reward-SQL: Boosting Text-to-SQL via Stepwise Execution-Aware Reasoning and Process-Supervised Rewardscited by 27
  2. 2025RL Grokking Recipe: How Does RL Unlock and Transfer New Algorithms in LLMs?cited by 23
  3. 2025DAComp: Benchmarking Data Agents across the Full Data Intelligence Lifecyclecited by 23
  4. 2025Enterprise Deep Research: Steerable Multi-Agent Deep Research for Enterprise Analyticscited by 20
  5. 2025Test-Time Adaptation for LLM Agents via Environment Interactioncited by 20
  6. 2025BIRD-INTERACT: Re-imagining Text-to-SQL Evaluation for Large Language Models via Lens of Dynamic Interactionscited by 19
  7. 2024TQA-Bench: Evaluating LLMs for Multi-Table Question Answeringcited by 17
  8. 2025Towards LLM Agents for Earth Observationcited by 17
  9. 2025Can LLMs Help You at Work? A Sandbox for Evaluating LLM Agents in Enterprise Environmentscited by 17
  10. 2025ELT-Bench: An End-to-End Benchmark for Evaluating AI Agents on ELT Pipelinescited by 16
  11. 2025SWE-SQL: Illuminating LLM Pathways to Solve User SQL Issues in Real-World Applicationscited by 16
  12. 2025EnrichIndex: Using LLMs to Enrich Retrieval Indices Offlinecited by 15
  13. 2025MARS-SQL: A multi-agent reinforcement learning framework for Text-to-SQLcited by 15
  14. 2025Think2SQL: Reinforce LLM Reasoning Capabilities for Text2SQLcited by 14
  15. 2026AgentSM: Semantic Memory for Agentic Text-to-SQLcited by 14
  16. 2026The Evolution of Tool Use in LLM Agents: From Single-Tool Call to Multi-Tool Orchestrationcited by 14
  17. 2026Pervasive Annotation Errors Break Text-to-SQL Benchmarks and Leaderboardscited by 13
  18. 2025Integrating Large Language Models for Automated Structural Analysiscited by 12
  19. 2025ExeSQL: Self-Taught Text-to-SQL Models with Execution-Driven Bootstrapping for SQL Dialectscited by 12
  20. 2025KNN-SSD: Enabling Dynamic Self-Speculative Decoding via Nearest Neighbor Layer Set Optimizationcited by 11
  21. 2025Scaling Generalist Data-Analytic Agentscited by 11
  22. 2024A Survey of Large Language Model-Based Generative AI for Text-to-SQL: Benchmarks, Applications, Use Cases, and Challengescited by 10
  23. 2025Tabular Data Understanding with LLMs: A Survey of Recent Advances and Challengescited by 10
  24. 2025Text-to-SQL as Dual-State Reasoning: Integrating Adaptive Context and Progressive Generationcited by 10
  25. 2025Knapsack Optimization-based Schema Linking for LLM-based Text-to-SQL Generationcited by 9
  26. 2025Multilingual Text-to-SQL: Benchmarking the Limits of Language Models with Collaborative Language Agentscited by 8
  27. 2026APEX-SQL: Talking to the data via Agentic Exploration for Text-to-SQLcited by 8
  28. 2026Mitigating Bias in Locally Constrained Decoding via Tractable Proposalscited by 8
  29. 2025GraphRAFT: Retrieval Augmented Fine-Tuning for Knowledge Graphs on Graph Databasescited by 7
  30. 2026Arming Data Agents with Tribal Knowledgecited by 7
  31. 2026Query-Aware Flow Diffusion for Graph-Based RAG with Retrieval Guaranteescited by 7
  32. 2026From Multi-Agent to Single-Agent: When Is Skill Distillation Beneficial?cited by 6
  33. 2025Text-to-Pipeline: Bridging Natural Language and Data Preparation Pipelinescited by 5
  34. 2025Benchmarking AI Models in Software Engineering: A Review, Search Tool, and Unified Approach for Elevating Benchmark Qualitycited by 4
  35. 2025Measuring Data Science Automation: A Survey of Evaluation Tools for AI Assistants and Agentscited by 4
  36. 2025Code Reasoning for Software Engineering Tasks: A Survey and A Call to Actioncited by 4
  37. 2025RubikSQL: Lifelong Learning Agentic Knowledge Base as an Industrial NL2SQL Systemcited by 4
  38. 2025SpotIt: Evaluating Text-to-SQL Evaluation with Formal Verificationcited by 4

50 most cited of 232 All 232 (JSON)