# Speak to your Parser: Interactive Text-to-SQL with Natural Language Feedback

arXiv: https://arxiv.org/abs/2005.02539 · HTML: https://mlsanity.com/paper/2005.02539/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 May 2020.

## Built on by (most cited first, 30 of 44)

- [Teaching Large Language Models to Self-Debug](https://mlsanity.com/paper/2304.05128/) (2023)
- [CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing](https://mlsanity.com/paper/2305.11738/) (2023)
- [Self-critiquing models for assisting human evaluators](https://mlsanity.com/paper/2206.05802/) (2022)
- [MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback](https://mlsanity.com/paper/2309.10691/) (2023)
- [REFINER: Reasoning Feedback on Intermediate Representations](https://mlsanity.com/paper/2304.01904/) (2023)
- [Training Language Models with Language Feedback at Scale](https://mlsanity.com/paper/2303.16755/) (2023)
- [Improving Code Generation by Training with Natural Language Feedback](https://mlsanity.com/paper/2303.16749/) (2023)
- [Unleashing the potential of prompt engineering for large language models](https://mlsanity.com/paper/2310.14735/) (2023)
- [Exploring Chain-of-Thought Style Prompting for Text-to-SQL](https://mlsanity.com/paper/2305.14215/) (2023)
- [Training Language Models with Language Feedback](https://mlsanity.com/paper/2204.14146/) (2022)
- [Recent Advances in Text-to-SQL: A Survey of What We Have and What We Expect](https://mlsanity.com/paper/2208.10099/) (2022)
- [EHRSQL: A Practical Text-to-SQL Benchmark for Electronic Health Records](https://mlsanity.com/paper/2301.07695/) (2023)
- [Text-to-SQL in the Wild: A Naturally-Occurring Dataset Based on Stack Exchange Data](https://mlsanity.com/paper/2106.05006/) (2021)
- [Benchmarking and Improving Text-to-SQL Generation under Ambiguity](https://mlsanity.com/paper/2310.13659/) (2023)
- [NL-EDIT: Correcting semantic parse errors through natural language interaction](https://mlsanity.com/paper/2103.14540/) (2021)
- [CHATREPORT: Democratizing Sustainability Disclosure Analysis through LLM-based Tools](https://mlsanity.com/paper/2307.15770/) (2023)
- [BlendSQL: A Scalable Dialect for Unifying Hybrid Question Answering in Relational Algebra](https://mlsanity.com/paper/2402.17882/) (2024)
- [Interactive Grounded Language Understanding in a Collaborative Environment: IGLU 2021](https://mlsanity.com/paper/2205.02388/) (2022)
- [Text-to-SQL Error Correction with Language Models of Code](https://mlsanity.com/paper/2305.13073/) (2023)
- [Interactive Text-to-SQL Generation via Editable Step-by-Step Explanations](https://mlsanity.com/paper/2305.07372/) (2023)
- [Learning to Simulate Natural Language Feedback for Interactive Semantic Parsing](https://mlsanity.com/paper/2305.08195/) (2023)
- [Learning Structural Edits via Incremental Tree Transformations](https://mlsanity.com/paper/2101.12087/) (2021)
- [NeurIPS 2021 Competition IGLU: Interactive Grounded Language Understanding in a Collaborative Environment](https://mlsanity.com/paper/2110.06536/) (2021)
- [IGLU 2022: Interactive Grounded Language Understanding in a Collaborative Environment at NeurIPS 2022](https://mlsanity.com/paper/2205.13771/) (2022)
- [Transforming Human-Centered AI Collaboration: Redefining Embodied Agents Capabilities through Interactive Grounded Language Instructions](https://mlsanity.com/paper/2305.10783/) (2023)
- [Non-Programmers Can Label Programs Indirectly via Active Examples: A Case Study with Text-to-SQL](https://mlsanity.com/paper/2205.12422/) (2022)
- [Hierarchical Neural Data Synthesis for Semantic Parsing](https://mlsanity.com/paper/2112.02212/) (2021)
- [SQLucid: Grounding Natural Language Database Queries with Interactive Explanations](https://mlsanity.com/paper/2409.06178/) (2024)
- [Towards Transparent Interactive Semantic Parsing via Step-by-Step Correction](https://mlsanity.com/paper/2110.08345/) (2021)
- [Interscript: A dataset for interactive learning of scripts through error feedback](https://mlsanity.com/paper/2112.07867/) (2021)

## Abstract (arXiv)

We study the task of semantic parse correction with natural language feedback. Given a natural language utterance, most semantic parsing systems pose the problem as one-shot translation where the utterance is mapped to a corresponding logical form. In this paper, we investigate a more interactive scenario where humans can further interact with the system by providing free-form natural language feedback to correct the system when it generates an inaccurate interpretation of an initial utterance. We focus on natural language to SQL systems and construct, SPLASH, a dataset of utterances, incorrect SQL interpretations and the corresponding natural language feedback. We compare various reference models for the correction task and show that incorporating such a rich form of feedback can significantly improve the overall semantic parsing accuracy while retaining the flexibility of natural language interaction. While we estimated human correction accuracy is 81.5%, our best model achieves only 25.1%, which leaves a large gap for improvement in future research. SPLASH is publicly available at https://aka.ms/Splash_dataset.