mlsanityData to 7 Oct 2026

Pervasive Annotation Errors Break Text-to-SQL Benchmarks and Leaderboards

arXiv 2601.08778 ↗First posted 13 Jan 2026cs.AI cs.DBcited by 12

Not on the map. Why?

Abstract (arXiv)

Researchers have proposed numerous text-to-SQL techniques to streamline data analytics and accelerate the development of data-driven applications. To compare these techniques and select the best one for deployment, the community depends on public benchmarks and their leaderboards. Since these benchmarks heavily rely on human annotations during question construction and answer evaluation, the validity of the annotations is crucial. In this paper, we conduct an empirical study that (i) benchmarks annotation error rates for two widely used text-to-SQL benchmarks, BIRD and Spider 2.0-Snow, and (ii) corrects a subset of the BIRD development (Dev) set to measure the impact of annotation errors on text-to-SQL agent performance and leaderboard rankings. Through expert analysis, we show that BIRD Mini-Dev and Spider 2.0-Snow have error rates of 52.8% and 62.8%, respectively. We re-evaluate all 16 open-source agents from the BIRD leaderboard on both the original and the corrected BIRD Dev subsets. We show that performance changes range from -7% to 31% (in relative terms) and rank changes range from -9 to +9 positions. We further assess whether these impacts generalize to the full BIRD Dev set. We find that the rankings of agents on the uncorrected subset correlate strongly with those on the full Dev set (Spearman's rs=0.85, p=3.26e-5), whereas they correlate weakly with those on the corrected subset (Spearman's rs=0.32, p=0.23). These findings show that annotation errors can significantly distort reported performance and rankings, potentially misguiding research directions or deployment choices. Our code and data are available at https://github.com/uiuc-kang-lab/texttosqlbenchmarks.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 37 of its 76 references that are in the map. First posted on arXiv 13 Jan 2026.

Builds on

37 of 76 references.

References · 37

  1. 2024Qwen2.5 Technical Reportcited by 4,713
  2. 2024A Survey on LLM-as-a-Judgecited by 1,384
  3. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  4. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  5. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,050
  6. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 667
  7. 2023DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correctioncited by 465
  8. 2023Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluationcited by 378
  9. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 362
  10. 2024Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflowscited by 223
  11. 2024AI Agents That Mattercited by 199
  12. 2024CodeS: Towards Building Open-source Language Models for Text-to-SQLcited by 183
25 more
  1. 2023MAC-SQL: A Multi-Agent Collaborative Framework for Text-to-SQLcited by 158
  2. 2021KaggleDBQA: Realistic Evaluation of Text-to-SQL Parserscited by 105
  3. 2017Learning a Neural Semantic Parser from User Feedbackcited by 98
  4. 2024SWE-Bench+: Enhanced Coding Benchmark for LLMscited by 95
  5. 2025Establishing Best Practices for Building Rigorous Agentic Benchmarkscited by 94
  6. 2025OmniSQL: Synthesizing High-quality Text-to-SQL Data at Scalecited by 90
  7. 2025Reasoning-SQL: Reinforcement Learning with SQL Tailored Partial Rewards for Reasoning-Enhanced Text-to-SQLcited by 59
  8. 2024RSL-SQL: Robust Schema Linking in Text-to-SQL Generationcited by 51
  9. 2025OpenSearch-SQL: Enhancing Text-to-SQL with Dynamic Few-shot and Consistency Alignmentcited by 51
  10. 2024Before Generation, Align it! A Novel and Effective Strategy for Mitigating Hallucinations in Text-to-SQL Generationcited by 49
  11. 2024DTS-SQL: Decomposed Text-to-SQL with Small Large Language Modelscited by 45
  12. 2025UTBoost: Rigorous Evaluation of Coding Agents on SWE-Benchcited by 44
  13. 2024BEAVER: An Enterprise Benchmark for Text-to-SQLcited by 37
  14. 2024E-SQL: Direct Schema Linking via Question Enrichment in Text-to-SQLcited by 35
  15. 2025ReFoRCE: A Text-to-SQL Agent with Self-Refinement, Consensus Enforcement, and Column Explorationcited by 34
  16. 2024Understanding the Effects of Noise in Text-to-SQL: An Examination of the BIRD-Bench Benchmarkcited by 33
  17. 2025Alpha-SQL: Zero-Shot Text-to-SQL using Monte Carlo Tree Searchcited by 33
  18. 2023Evaluating Cross-Domain Text-to-SQL Models and Benchmarkscited by 32
  19. 2024Comparing Two Model Designs for Clinical Note Generation; Is an LLM a Useful Evaluator of Consistency?cited by 20
  20. 2024Leveraging Large Language Models for Relevance Judgments in Legal Case Retrievalcited by 16
  21. 2024Evaluating Software Development Agents: Patch Patterns, Code Quality, and Issue Complexity in Real-World GitHub Scenarioscited by 16
  22. 2024Mixing It Up: The Cocktail Effect of Multi-Task Fine-Tuning on LLM Performance -- A Case Study in Financecited by 15
  23. 2025CSC-SQL: Corrective Self-Consistency in Text-to-SQL via Reinforcement Learningcited by 15
  24. 2024Towards Leveraging Large Language Models for Automated Medical Q&A Evaluationcited by 14
  25. 2025Cheaper, Better, Faster, Stronger: Robust Text-to-SQL without Chain-of-Thought or Fine-Tuningcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 12.

12 papers.

  1. 2026Arming Data Agents with Tribal Knowledgecited by 6
  2. 2026Noisy Data is Destructive to Reinforcement Learning with Verifiable Rewardscited by 4
  3. 2026ELT-Bench-Verified: Benchmark Quality Issues Underestimate AI Agent Capabilitiescited by 2
  4. 2026Tool-Making and Self-Evolving LLM Agents in Low-Latency Systemscited by 2
  5. 2026Generating Expressive and Customizable Evals for Timeseries Data Analysis Agents with AgentFuelcited by 1
  6. 2026Human-Level Text-to-SQL via Reinforcement Learning on Verified Data, Without Pipeline Engineeringcited by 1
  7. 2026How Far Do On-Prem Open LLMs Get on Text-to-SQL? A Cross-Family Size x Technique Frontier on BIRDcited by 0
  8. 2026Agentic Data Environmentscited by 0
  9. 2026Iteration Without Elaboration: A Simple ReAct Architecture Suffices for Text-to-SQL Generationcited by 0
  10. 2026Certified Against Which Oracle? Execution Labels Set the Reported Risk of Conformal Abstention for Text-to-SQLcited by 0
  11. 2026BudgetSchemaBench: A Budget-Swept Diagnostic for Schema Context in Text-to-SQLcited by 0
  12. 2026Argo-Bench: Evaluating Data Agents on Enterprise-Scale Workflowscited by 0