mlsanityData to 7 Oct 2026

Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task

arXiv 1809.08887 ↗First posted 24 Sep 2018cs.CL cs.AIcited by 1,120

Not on the map. Why?

Cited by 121 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present Spider, a large-scale, complex and cross-domain semantic parsing and text-to-SQL dataset annotated by 11 college students. It consists of 10,181 questions and 5,693 unique complex SQL queries on 200 databases with multiple tables, covering 138 different domains. We define a new complex and cross-domain semantic parsing and text-to-SQL task where different complex SQL queries and databases appear in train and test sets. In this way, the task requires the model to generalize well to both new SQL queries and new database schemas. Spider is distinct from most of the previous semantic parsing tasks because they all use a single database and the exact same programs in the train set and the test set. We experiment with various state-of-the-art models and the best model achieves only 12.4% exact matching accuracy on a database split setting. This shows that Spider presents a strong challenge for future research. Our dataset and task are publicly available at https://yale-lily.github.io/spider

Key facts. Cited by 1,120 papers in the map (data through 7 Oct 2026). First posted on arXiv 24 Sep 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 75; 2022: 96; 2023: 151; 2024: 228; 2025: 270; 2026: 300.

1,120 papers.

Sort:

By area:

  1. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  2. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  3. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  4. 2024Qwen2.5-Coder Technical Reportcited by 1,151
  5. 2023Teaching Large Language Models to Self-Debugcited by 963
  6. 2021Measuring Coding Challenge Competence With APPScited by 879
  7. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 667
  8. 2024A Survey on Large Language Models for Code Generationcited by 646
  9. 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 552
  10. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  11. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 491
  12. 2023DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correctioncited by 465
38 more
  1. 2021MetaICL: Learning to Learn In Contextcited by 450
  2. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 450
  3. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  4. 2021TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Financecited by 402
  5. 2021PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Modelscited by 402
  6. 2025LLMs Get Lost In Multi-Turn Conversationcited by 400
  7. 2021QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answeringcited by 386
  8. 2023Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluationcited by 378
  9. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 347
  10. 2023StructGPT: A General Framework for Large Language Model to Reason over Structured Datacited by 339
  11. 2023How is ChatGPT's behavior changing over time?cited by 265
  12. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 235
  13. 2021TAPEX: Table Pre-training via Learning a Neural SQL Executorcited by 233
  14. 2022UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Modelscited by 231
  15. 2023InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedbackcited by 229
  16. 2023LEVER: Learning to Verify Language-to-Code Generation with Executioncited by 228
  17. 2022Binding Language Models in Symbolic Languagescited by 224
  18. 2024Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflowscited by 223
  19. 2023RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQLcited by 191
  20. 2025Survey on Evaluation of LLM-based Agentscited by 190
  21. 2024CodeS: Towards Building Open-source Language Models for Text-to-SQLcited by 183
  22. 2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 179
  23. 2021FeTaQA: Free-form Table Question Answeringcited by 174
  24. 2023SantaCoder: don't reach for the stars!cited by 169
  25. 2024CHESS: Contextual Harnessing for Efficient SQL Synthesiscited by 164
  26. 2021Constrained Language Models Yield Few-Shot Semantic Parserscited by 162
  27. 2023C3: Zero-shot Text-to-SQL with ChatGPTcited by 159
  28. 2023MAC-SQL: A Multi-Agent Collaborative Framework for Text-to-SQLcited by 158
  29. 2024Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Surveycited by 157
  30. 2022Synchromesh: Reliable code generation from pre-trained language modelscited by 156
  31. 2021Episodic Transformer for Vision-and-Language Navigationcited by 155
  32. 2025LLaDA2.0: Scaling Up Diffusion Language Models to 100Bcited by 154
  33. 2022Instruction Induction: From Few Examples to Natural Language Task Descriptionscited by 152
  34. 2021CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLPcited by 142
  35. 2024Multi-Step Reasoning with Large Language Models, a Surveycited by 140
  36. 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 138
  37. 2022MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generationcited by 132
  38. 2024CHASE-SQL: Multi-Path Reasoning and Preference Optimized Candidate Selection in Text-to-SQLcited by 132

50 most cited of 1,120 All 1,120 (JSON)