# CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databases

arXiv: https://arxiv.org/abs/1909.05378 · HTML: https://mlsanity.com/paper/1909.05378/

## Key facts

- Cited by 185 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Sep 2019.

## Built on by (most cited first, 30 of 185)

- [Teaching Large Language Models to Self-Debug](https://mlsanity.com/paper/2304.05128/) (2023)
- [CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis](https://mlsanity.com/paper/2203.13474/) (2022)
- [PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models](https://mlsanity.com/paper/2109.05093/) (2021)
- [InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback](https://mlsanity.com/paper/2306.14898/) (2023)
- [UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models](https://mlsanity.com/paper/2201.05966/) (2022)
- [MapCoder: Multi-Agent Code Generation for Competitive Problem Solving](https://mlsanity.com/paper/2405.11403/) (2024)
- [Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Survey](https://mlsanity.com/paper/2402.17944/) (2024)
- [Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQL](https://mlsanity.com/paper/2406.08426/) (2024)
- [A comprehensive evaluation of ChatGPT's zero-shot Text-to-SQL capability](https://mlsanity.com/paper/2303.13547/) (2023)
- [Natural Language to Code Generation in Interactive Data Science Notebooks](https://mlsanity.com/paper/2212.09248/) (2022)
- [RASAT: Integrating Relational Structures into Pretrained Seq2Seq Model for Text-to-SQL](https://mlsanity.com/paper/2205.06983/) (2022)
- [Is GPT-4 a Good Data Analyst?](https://mlsanity.com/paper/2305.15038/) (2023)
- [A Survey on Employing Large Language Models for Text-to-SQL Tasks](https://mlsanity.com/paper/2407.15186/) (2024)
- [ACT-SQL: In-Context Learning for Text-to-SQL with Automatically-Generated Chain-of-Thought](https://mlsanity.com/paper/2310.17342/) (2023)
- [Few-Shot Bot: Prompt-Based Learning for Dialogue Systems](https://mlsanity.com/paper/2110.08118/) (2021)
- [Learning to Clarify: Multi-turn Conversations with Action-Based Contrastive Self-Training](https://mlsanity.com/paper/2406.00222/) (2024)
- [A Survey of Text-to-SQL in the Era of LLMs: Where are we, and where are we going?](https://mlsanity.com/paper/2408.05109/) (2024)
- [Rethinking Explainability as a Dialogue: A Practitioner's Perspective](https://mlsanity.com/paper/2202.01875/) (2022)
- [A Survey on Text-to-SQL Parsing: Concepts, Methods, and Future Directions](https://mlsanity.com/paper/2208.13629/) (2022)
- [Recent Advances in Text-to-SQL: A Survey of What We Have and What We Expect](https://mlsanity.com/paper/2208.10099/) (2022)
- [StreamBench: Towards Benchmarking Continuous Improvement of Language Agents](https://mlsanity.com/paper/2406.08747/) (2024)
- [Enhancing Few-shot Text-to-SQL Capabilities of Large Language Models: A Study on Prompt Design Strategies](https://mlsanity.com/paper/2305.12586/) (2023)
- [Neural Approaches to Conversational Information Retrieval](https://mlsanity.com/paper/2201.05176/) (2022)
- [SQL-PaLM: Improved Large Language Model Adaptation for Text-to-SQL (extended)](https://mlsanity.com/paper/2306.00739/) (2023)
- [Symbol-LLM: Towards Foundational Symbol-centric Interface For Large Language Models](https://mlsanity.com/paper/2311.09278/) (2023)
- [TabSQLify: Enhancing Reasoning Capabilities of LLMs Through Table Decomposition](https://mlsanity.com/paper/2404.10150/) (2024)
- [SPACE-3: Unified Dialog Model Pre-training for Task-Oriented Dialog Understanding and Generation](https://mlsanity.com/paper/2209.06664/) (2022)
- [NL-EDIT: Correcting semantic parse errors through natural language interaction](https://mlsanity.com/paper/2103.14540/) (2021)
- [Calibrated Interpretation: Confidence Estimation in Semantic Parsing](https://mlsanity.com/paper/2211.07443/) (2022)
- [CODESIM: Multi-Agent Code Generation and Problem Solving through Simulation-Driven Planning and Debugging](https://mlsanity.com/paper/2502.05664/) (2025)

## Abstract (arXiv)

We present CoSQL, a corpus for building cross-domain, general-purpose database (DB) querying dialogue systems. It consists of 30k+ turns plus 10k+ annotated SQL queries, obtained from a Wizard-of-Oz (WOZ) collection of 3k dialogues querying 200 complex DBs spanning 138 domains. Each dialogue simulates a real-world DB query scenario with a crowd worker as a user exploring the DB and a SQL expert retrieving answers with SQL, clarifying ambiguous questions, or otherwise informing of unanswerable questions. When user questions are answerable by SQL, the expert describes the SQL and execution results to the user, hence maintaining a natural interaction flow. CoSQL introduces new challenges compared to existing task-oriented dialogue datasets:(1) the dialogue states are grounded in SQL, a domain-independent executable representation, instead of domain-specific slot-value pairs, and (2) because testing is done on unseen databases, success requires generalizing to new domains. CoSQL includes three tasks: SQL-grounded dialogue state tracking, response generation from query results, and user dialogue act prediction. We evaluate a set of strong baselines for each task and show that CoSQL presents significant challenges for future research. The dataset, baselines, and leaderboard will be released at https://yale-lily.github.io/cosql.