mlsanityData to 14 Oct 2026

Dr.Spider: A Diagnostic Evaluation Benchmark towards Text-to-SQL Robustness

arXiv 2301.08881 ↗First posted 21 Jan 2023cs.CLcited by 53

Not on the map. Why?

Abstract (arXiv)

Neural text-to-SQL models have achieved remarkable performance in translating natural language questions into SQL queries. However, recent studies reveal that text-to-SQL models are vulnerable to task-specific perturbations. Previous curated robustness test sets usually focus on individual phenomena. In this paper, we propose a comprehensive robustness benchmark based on Spider, a cross-domain text-to-SQL benchmark, to diagnose the model robustness. We design 17 perturbations on databases, natural language questions, and SQL queries to measure the robustness from different angles. In order to collect more diversified natural question perturbations, we utilize large pretrained language models (PLMs) to simulate human behaviors in creating natural questions. We conduct a diagnostic study of the state-of-the-art models on the robustness set. Experimental results reveal that even the most robust model suffers from a 14.0% performance drop overall and a 50.7% performance drop on the most challenging perturbation. We also present a breakdown analysis regarding text-to-SQL model designs and provide insights for improving model robustness.

Key facts. Cited by 53 papers in the map (data through 14 Oct 2026). Builds on 45 of its 57 references that are in the map. First posted on arXiv 21 Jan 2023.

Builds on

45 of 57 references.

References · 45

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  10. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  11. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  12. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
33 more
  1. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,139
  2. 2018Self-Attention with Relative Position Representationscited by 1,120
  3. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
  4. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 807
  5. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 762
  6. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 687
  7. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 571
  8. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 541
  9. 2019Learning the Difference that Makes a Difference with Counterfactually-Augmented Datacited by 439
  10. 2019RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parserscited by 424
  11. 2021PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Modelscited by 411
  12. 2018Adversarial Example Generation with Syntactically Controlled Paraphrase Networkscited by 286
  13. 2022CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilitiescited by 249
  14. 2021Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Modelscited by 232
  15. 2022WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creationcited by 202
  16. 2021Generating Datasets with Pretrained Language Modelscited by 193
  17. 2020GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsingcited by 171
  18. 2020Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both?cited by 147
  19. 2020Bridging Textual and Tabular Data for Cross-Domain Text-to-SQL Semantic Parsingcited by 133
  20. 2020Structure-Grounded Pretraining for Text-to-SQLcited by 126
  21. 2021Towards Robustness of Text-to-SQL Models against Synonym Substitutioncited by 110
  22. 2021Robustness Gym: Unifying the NLP Evaluation Landscapecited by 108
  23. 2021KaggleDBQA: Realistic Evaluation of Text-to-SQL Parserscited by 106
  24. 2022Evaluating the Text-to-SQL Capabilities of Large Language Modelscited by 101
  25. 2020SmBoP: Semi-autoregressive Bottom-up Semantic Parsingcited by 97
  26. 2021Exploring Underexplored Limitations of Cross-Domain Text-to-SQL Generalizationcited by 92
  27. 2020On the Potential of Lexico-logical Alignments for Semantic Parsing to SQL Queriescited by 60
  28. 2021Neural Data Augmentation via Example Extrapolationcited by 54
  29. 2021Prompting Contrastive Explanations for Commonsense Reasoning Taskscited by 45
  30. 2022Quality Controlled Paraphrase Generationcited by 31
  31. 2022Towards Robustness of Text-to-SQL Models Against Natural and Realistic Adversarial Table Perturbationcited by 28
  32. 2020DuoRAT: Towards Simpler Text-to-SQL Modelscited by 17
  33. 2021On Robustness of Neural Semantic Parserscited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 10; 2024: 11; 2025: 13; 2026: 19.

53 papers.

Sort:

By area:

  1. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 688
  2. 2024CodeS: Towards Building Open-source Language Models for Text-to-SQLcited by 187
  3. 2024Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQLcited by 120
  4. 2025Measuring what Matters: Construct Validity in Large Language Model Benchmarkscited by 119
  5. 2025OmniSQL: Synthesizing High-quality Text-to-SQL Data at Scalecited by 91
  6. 2023Exploring Chain-of-Thought Style Prompting for Text-to-SQLcited by 69
  7. 2024A Survey on Employing Large Language Models for Text-to-SQL Taskscited by 67
  8. 2023How to Prompt LLMs for Text-to-SQL: A Study in Zero-shot, Single-domain, and Cross-domain Settingscited by 58
  9. 2024A Survey of Text-to-SQL in the Era of LLMs: Where are we, and where are we going?cited by 56
  10. 2024Large Language Model for Table Processing: A Surveycited by 52
  11. 2023Skill-Based Few-Shot Selection for In-Context Learningcited by 46
  12. 2023RobuT: A Systematic Study of Table QA Robustness Against Human-Annotated Adversarial Perturbationscited by 43
38 more
  1. 2023Benchmarking and Improving Text-to-SQL Generation under Ambiguitycited by 34
  2. 2023Natural Language Interfaces for Tabular Data Querying and Visualization: A Surveycited by 27
  3. 2023Interleaving Pre-Trained Language Models and Large Language Models for Zero-Shot NL2SQL Generationcited by 24
  4. 2023Selective Demonstrations for Cross-domain Text-to-SQLcited by 21
  5. 2023Observatory: Characterizing Embeddings of Relational Tablescited by 12
  6. 2025ExeSQL: Self-Taught Text-to-SQL Models with Execution-Driven Bootstrapping for SQL Dialectscited by 12
  7. 2024You Only Read Once (YORO): Learning to Internalize Database Knowledge for Text-to-SQLcited by 11
  8. 2024Solid-SQL: Enhanced Schema-linking based In-context Learning for Robust Text-to-SQLcited by 11
  9. 2025Confidence Estimation for Error Detection in Text-to-SQL Systemscited by 11
  10. 2024Auto-Cypher: Improving LLMs on Cypher generation via LLM-supervised generation-verification frameworkcited by 10
  11. 2025Tabular Data Understanding with LLMs: A Survey of Recent Advances and Challengescited by 10
  12. 2026EvoSchema: Towards Text-to-SQL Robustness Against Schema Evolutioncited by 5
  13. 2024Improving Generalization in Semantic Parsing by Increasing Natural Language Variationcited by 4
  14. 2024Mastering the Craft of Data Synthesis for CodeLLMscited by 3
  15. 2025M3TQA: Massively Multilingual Multitask Table Question Answeringcited by 3
  16. 2025Structural Deep Encoding for Table Question Answeringcited by 2
  17. 2025Evaluating NL2SQL via SQL2NLcited by 2
  18. 2024Relational Database Augmented Large Language Modelcited by 1
  19. 2025SQLSpace: A Representation Space for Text-to-SQL to Discover and Mitigate Robustness Gapscited by 1
  20. 2025RIDE: Difficulty Evolving Perturbation with Item Response Theory for Mathematical Reasoningcited by 1
  21. 2025Beyond SELECT: A Comprehensive Taxonomy-Guided Benchmark for Real-World Text-to-SQL Translationcited by 1
  22. 2026Mitigating Prompt-Induced Hallucinations in Large Language Models via Structured Reasoningcited by 1
  23. 2026KDCM: Reducing Hallucination in LLMs through Explicit Reasoning Structurescited by 1
  24. 2026SQLStructEval: Structural Evaluation of LLM Text-to-SQL Generationcited by 1
  25. 2026NL2SQLBench: A Modular Benchmarking Framework for LLM-Enabled NL2SQL Solutionscited by 1
  26. 2026EGREFINE: An Execution-Grounded Optimization Framework for Text-to-SQL Schema Refinementcited by 1
  27. 2026EntSQL: A Benchmark for Grounding Text-to-SQL in Long-Context Enterprise Knowledgecited by 1
  28. 2025GEMMA-SQL: A Novel Text-to-SQL Model Based on Large Language Modelscited by 0
  29. 2025A Multi-agent Text2SQL Framework using Small Language Models and Execution Feedbackcited by 0
  30. 2026Boundary-Aware NL2SQL: Integrating Reliability through Hybrid Reward and Data Synthesiscited by 0
  31. 2026FINER-SQL: Boosting Small Language Models for Text-to-SQLcited by 0
  32. 2026BADGER: Bridging Agentic and Deterministic Evaluation for Generative Enterprise Reasoningcited by 0
  33. 2026Beyond Static Rules: Automated Discovery of Latent Vulnerabilities in Text-to-SQLcited by 0
  34. 2026TrustDABench: Benchmarking Reliability and Robustness of LLMs for Structured Data Analysiscited by 0
  35. 2026DRL: A Deterministic Relational Middleware Layer for Transaction-Safe Enterprise NL2SQL Under Schema-Graph Scalingcited by 0
  36. 2026BIRD-History: A Benchmark for History-Driven Text-to-SQL with Fine-Grained Knowledge Annotationscited by 0
  37. 2026Replacing Training with Memory: Listwise Selection for Text-to-SQLcited by 0
  38. 2026TQTS-Bench: A Multi-Syntax Benchmark for Text-to-Query over Time-Series Databasescited by 0

50 most cited of 53 All 53 (JSON)