mlsanityData to 7 Oct 2026

Understanding the Effects of Noise in Text-to-SQL: An Examination of the BIRD-Bench Benchmark

arXiv 2402.12243 ↗First posted 19 Feb 2024cs.CLcited by 33

Not on the map. Why?

Abstract (arXiv)

Text-to-SQL, which involves translating natural language into Structured Query Language (SQL), is crucial for enabling broad access to structured databases without expert knowledge. However, designing models for such tasks is challenging due to numerous factors, including the presence of 'noise,' such as ambiguous questions and syntactical errors. This study provides an in-depth analysis of the distribution and types of noise in the widely used BIRD-Bench benchmark and the impact of noise on models. While BIRD-Bench was created to model dirty and noisy database values, it was not created to contain noise and errors in the questions and gold queries. We found that noise in questions and gold queries are prevalent in the dataset, with varying amounts across domains, and with an uneven distribution between noise types. The presence of incorrect gold SQL queries, which then generate incorrect gold answers, has a significant impact on the benchmark's reliability. Surprisingly, when evaluating models on corrected SQL queries, zero-shot baselines surpassed the performance of state-of-the-art prompting methods. We conclude that informative noise labels and reliable benchmarks are crucial to developing new Text-to-SQL methods that can handle varying types of noise. All datasets, annotations, and code are available at https://github.com/niklaswretblad/the-effects-of-noise-in-text-to-SQL.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Builds on 10 of its 13 references that are in the map. First posted on arXiv 19 Feb 2024.

Builds on

10 of 13 references.

References · 10

  1. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  2. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 679
  3. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 667
  4. 2023DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correctioncited by 465
  5. 2023MAC-SQL: A Multi-Agent Collaborative Framework for Text-to-SQLcited by 158
  6. 2021Towards Robustness of Text-to-SQL Models against Synonym Substitutioncited by 110
  7. 2021KaggleDBQA: Realistic Evaluation of Text-to-SQL Parserscited by 105
  8. 2021Exploring Underexplored Limitations of Cross-Domain Text-to-SQL Generalizationcited by 92
  9. 2023SQL-PaLM: Improved Large Language Model Adaptation for Text-to-SQL (extended)cited by 42
  10. 2023Evaluating Cross-Domain Text-to-SQL Models and Benchmarkscited by 32

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 14; 2026: 16.

33 papers.

Sort:

By area:

  1. 2025Establishing Best Practices for Building Rigorous Agentic Benchmarkscited by 94
  2. 2025ELT-Bench: An End-to-End Benchmark for Evaluating AI Agents on ELT Pipelinescited by 15
  3. 2025Understanding, Detecting, and Repairing Real-World In-Context-Learning-Based Text-to-SQL Errorscited by 14
  4. 2024ROUTE: Robust Multitask Tuning and Collaboration for Text-to-SQLcited by 13
  5. 2026Pervasive Annotation Errors Break Text-to-SQL Benchmarks and Leaderboardscited by 12
  6. 2024FLEX: Expert-level False-Less EXecution Metric for Reliable Text-to-SQL Benchmarkcited by 11
  7. 2025Confidence Estimation for Error Detection in Text-to-SQL Systemscited by 10
  8. 2025AmbiSQL: Interactive Ambiguity Detection and Resolution for Text-to-SQLcited by 10
  9. 2025LogicCat: A Chain-of-Thought Text-to-SQL Benchmark for Complex Reasoningcited by 7
  10. 2025Can LLMs Narrate Tabular Data? An Evaluation Framework for Natural Language Representations of Text-to-SQL System Outputscited by 7
  11. 2025CoddLLM: Empowering Large Language Models for Data Analyticscited by 6
  12. 2025SAFE-SQL: Self-Augmented In-Context Learning with Fine-grained Example Selection for Text-to-SQLcited by 4
21 more
  1. 2025Exploring the Landscape of Text-to-SQL with Large Language Models: Progresses, Challenges and Opportunitiescited by 4
  2. 2026Noisy Data is Destructive to Reinforcement Learning with Verifiable Rewardscited by 4
  3. 2025SpotIt: Evaluating Text-to-SQL Evaluation with Formal Verificationcited by 3
  4. 2026ClaimDB: A Fact Verification Benchmark over Large Structured Datacited by 3
  5. 2026Semantic Layers for Reliable LLM-Powered Data Analytics: A Paired Benchmark of Accuracy and Hallucination Across Three Frontier Modelscited by 3
  6. 2024Multi-Turn Interactions for Text-to-SQL with Large Language Modelscited by 2
  7. 2025Confidence Estimation for Text-to-SQL in Large Language Modelscited by 2
  8. 2026ELT-Bench-Verified: Benchmark Quality Issues Underestimate AI Agent Capabilitiescited by 2
  9. 2026AgentNLQ: A General-Purpose Agent for Natural Language to SQLcited by 2
  10. 2026Tool-Making and Self-Evolving LLM Agents in Low-Latency Systemscited by 2
  11. 2026Human-Level Text-to-SQL via Reinforcement Learning on Verified Data, Without Pipeline Engineeringcited by 1
  12. 2025LLMSQL: Upgrading WikiSQL for the LLM Era of Text-to-SQLcited by 0
  13. 2025DCMM-SQL: Automated Data-Centric Pipeline and Multi-Model Collaboration Training for Text-to-SQL Modelcited by 0
  14. 2026IESR:Efficient MCTS-Based Modular Reasoning for Text-to-SQL with Large Language Modelscited by 0
  15. 2026Data Intelligence Agents: Interpreting, Modeling, and Querying Enterprise Data via Autonomous Coding Agentscited by 0
  16. 2026Towards a Systems Foundation for Agentic Skills: Architecture, Lifecycle, and Securitycited by 0
  17. 2026BI-Agent and BI-Bench: Towards Automating End-to-End Business Intelligencecited by 0
  18. 2026Certified Against Which Oracle? Execution Labels Set the Reported Risk of Conformal Abstention for Text-to-SQLcited by 0
  19. 2026Auditing and Repairing LLM-as-Judge Failures in a Production Text-to-SQL Pipelinecited by 0
  20. 2026BudgetSchemaBench: A Budget-Swept Diagnostic for Schema Context in Text-to-SQLcited by 0
  21. 2026Argo-Bench: Evaluating Data Agents on Enterprise-Scale Workflowscited by 0