mlsanityData to 14 Oct 2026

BIRD-History: A Benchmark for History-Driven Text-to-SQL with Fine-Grained Knowledge Annotations

arXiv 2608.29345 ↗First posted 29 Aug 2026cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

While recent Large Language Model (LLM)-based text-to-SQL systems achieve impressive performance on standard benchmarks, they struggle when user queries implicitly rely on domain-specific knowledge, such as business logic, data conventions, and analytical practices, that is neither captured by the schema nor explicitly stated in the natural language question. Historical SQL query logs offer a valuable source of such knowledge, yet existing benchmarks do not adequately support evaluation of history-driven approaches. To address this gap, we introduce BIRD-History, a benchmark consisting of 1,393 tasks across 11 databases, designed to evaluate text-to-SQL systems' ability to ground underspecified natural language questions using historical SQL scripts. Each task is annotated with ground-truth labels specifying which historical queries contain relevant knowledge and which SQL clauses encode it, enabling systematic evaluation of both retrieval effectiveness and knowledge utilization. Alongside the benchmark, we propose a plug-in retriever that extracts five types of external knowledge from historical SQL scripts, then retrieves and reranks relevant fragments for query generation. The retriever integrates seamlessly into existing few-shot text-to-SQL pipelines without requiring prompt modifications. Experiments demonstrate consistent improvements across four text-to-SQL systems, highlighting the value of leveraging historical query logs for handling underspecified queries. Dataset and code are open-sourced on https://github.com/zjuidg/BIRD-History.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 34 of its 69 references that are in the map. First posted on arXiv 29 Aug 2026.

Builds on

34 of 69 references.

References · 34

  1. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,139
  2. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 688
  3. 2023DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correctioncited by 477
  4. 2019RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parserscited by 424
  5. 2020TaBERT: Pretraining for Joint Understanding of Textual and Tabular Datacited by 398
  6. 2023Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluationcited by 385
  7. 2024Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflowscited by 232
  8. 2019Towards Complex Text-to-SQL in Cross-Domain Database with Intermediate Representationcited by 210
  9. 2023RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQLcited by 195
  10. 2024CodeS: Towards Building Open-source Language Models for Text-to-SQLcited by 187
  11. 2024CHESS: Contextual Harnessing for Efficient SQL Synthesiscited by 169
  12. 2023MAC-SQL: A Multi-Agent Collaborative Framework for Text-to-SQLcited by 164
22 more
  1. 2020Bridging Textual and Tabular Data for Cross-Domain Text-to-SQL Semantic Parsingcited by 133
  2. 2024Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQLcited by 120
  3. 2018Coarse-to-Fine Decoding for Neural Semantic Parsingcited by 110
  4. 2021Towards Robustness of Text-to-SQL Models against Synonym Substitutioncited by 110
  5. 2021KaggleDBQA: Realistic Evaluation of Text-to-SQL Parserscited by 106
  6. 2024MCS-SQL: Leveraging Multiple Prompts and Multiple-Choice Selection For Text-to-SQL Generationcited by 105
  7. 2020SmBoP: Semi-autoregressive Bottom-up Semantic Parsingcited by 97
  8. 2021Exploring Underexplored Limitations of Cross-Domain Text-to-SQL Generalizationcited by 92
  9. 2025OmniSQL: Synthesizing High-quality Text-to-SQL Data at Scalecited by 91
  10. 2019Representing Schema Structure with Graph Neural Networks for Text-to-SQL Parsingcited by 74
  11. 2025SQL-R1: Training Natural Language to SQL Reasoning Model By Reinforcement Learningcited by 68
  12. 2023Dr.Spider: A Diagnostic Evaluation Benchmark towards Text-to-SQL Robustnesscited by 53
  13. 2025OpenSearch-SQL: Enhancing Text-to-SQL with Dynamic Few-shot and Consistency Alignmentcited by 53
  14. 2025XiYan-SQL: A Novel Multi-Generator Framework For Text-to-SQLcited by 40
  15. 2023Towards Knowledge-Intensive Text-to-SQL Semantic Parsing with Formulaic Knowledgecited by 18
  16. 2025SHARE: An SLM-based Hierarchical Action CorREction Assistant for Text-to-SQLcited by 16
  17. 2025Cheaper, Better, Faster, Stronger: Robust Text-to-SQL without Chain-of-Thought or Fine-Tuningcited by 13
  18. 2025SLM-SQL: An Exploration of Small Language Models for Text-to-SQLcited by 10
  19. 2025Uncovering the Impact of Chain-of-Thought Reasoning for Direct Preference Optimization: Lessons from Text-to-SQLcited by 9
  20. 2025PARROT: A Benchmark for Evaluating LLMs in Cross-System SQL Translationcited by 5
  21. 2025DCG-SQL: Enhancing In-Context Learning for Text-to-SQL with Deep Contextual Schema Link Graphcited by 4
  22. 2025Feather-SQL: A Lightweight NL2SQL Framework with Dual-Model Collaboration Paradigm for Small Language Modelscited by 2

Built on by

Not cited yet.