# Confidence Estimation for Error Detection in Text-to-SQL Systems

arXiv: https://arxiv.org/abs/2501.09527 · HTML: https://mlsanity.com/paper/2501.09527/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 40 references that are in the map.
- First posted on arXiv 16 Jan 2025.

## Builds on (most cited first, 20 of 20)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task](https://mlsanity.com/paper/1809.08887/) (2018)
- [Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs](https://mlsanity.com/paper/2305.03111/) (2023)
- [Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation](https://mlsanity.com/paper/2308.15363/) (2023)
- [Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning](https://mlsanity.com/paper/2002.06470/) (2020)
- [Out-of-Distribution Detection and Selective Generation for Conditional Language Models](https://mlsanity.com/paper/2209.15558/) (2022)
- [Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both?](https://mlsanity.com/paper/2010.12725/) (2020)
- [KaggleDBQA: Realistic Evaluation of Text-to-SQL Parsers](https://mlsanity.com/paper/2106.11455/) (2021)
- [Dr.Spider: A Diagnostic Evaluation Benchmark towards Text-to-SQL Robustness](https://mlsanity.com/paper/2301.08881/) (2023)
- [Understanding the Effects of Noise in Text-to-SQL: An Examination of the BIRD-Bench Benchmark](https://mlsanity.com/paper/2402.12243/) (2024)
- [Confidence Modeling for Neural Semantic Parsing](https://mlsanity.com/paper/1805.04604/) (2018)
- [Calibrated Interpretation: Confidence Estimation in Semantic Parsing](https://mlsanity.com/paper/2211.07443/) (2022)
- [TrustSQL: Benchmarking Text-to-SQL Reliability with Penalty-Based Scoring](https://mlsanity.com/paper/2403.15879/) (2024)
- [Error Detection for Text-to-SQL Semantic Parsing](https://mlsanity.com/paper/2305.13683/) (2023)
- [SUN: Exploring Intrinsic Uncertainties in Text-to-SQL Parsers](https://mlsanity.com/paper/2209.06442/) (2022)
- [ProbGate at EHRSQL 2024: Enhancing SQL Query Generation Accuracy through Probabilistic Threshold Filtering and Error Handling](https://mlsanity.com/paper/2404.16659/) (2024)
- [Towards Unbiased Evaluation of Detecting Unanswerable Questions in EHRSQL](https://mlsanity.com/paper/2405.01588/) (2024)

## Built on by (most cited first, 11 of 11)

- [Confidence Estimation for Text-to-SQL in Large Language Models](https://mlsanity.com/paper/2508.14056/) (2025)
- [Node-Level Uncertainty Estimation in LLM-Generated SQL](https://mlsanity.com/paper/2511.13984/) (2025)
- [Team Anotheroption at SemEval-2025 Task 8: Bridging the Gap Between Open-Source and Proprietary LLMs in Table QA](https://mlsanity.com/paper/2506.09657/) (2025)
- [The benefits of query-based KGQA systems for complex and temporal questions in LLM era](https://mlsanity.com/paper/2507.11954/) (2025)
- [SCP-NL2TL: Selective Conformal Prediction with Semantic Verification for Natural Language to Temporal Logic Specifications](https://mlsanity.com/paper/2608.05439/) (2026)
- [Never the Number: Structural Abstention for AI Systems Whose Answers Are Consumed as Fact](https://mlsanity.com/paper/2608.13926/) (2026)
- [SCARE: A Benchmark for SQL Correction and Question Answerability Classification for Reliable EHR Question Answering](https://mlsanity.com/paper/2511.17559/) (2025)
- [Bridging Global Intent with Local Details: A Hierarchical Representation Approach for Semantic Validation in Text-to-SQL](https://mlsanity.com/paper/2512.22744/) (2025)
- [TrustDABench: Benchmarking Reliability and Robustness of LLMs for Structured Data Analysis](https://mlsanity.com/paper/2608.24145/) (2026)
- [Certified Against Which Oracle? Execution Labels Set the Reported Risk of Conformal Abstention for Text-to-SQL](https://mlsanity.com/paper/2609.25938/) (2026)
- [Trajectory-Derived Confidence for Reliable, Resource-Aware Clinical Text-to-SQL Agents](https://mlsanity.com/paper/2610.04156/) (2026)

## Abstract (arXiv)

Text-to-SQL enables users to interact with databases through natural language, simplifying the retrieval and synthesis of information. Despite the success of large language models (LLMs) in converting natural language questions into SQL queries, their broader adoption is limited by two main challenges: achieving robust generalization across diverse queries and ensuring interpretative confidence in their predictions. To tackle these issues, our research investigates the integration of selective classifiers into Text-to-SQL systems. We analyse the trade-off between coverage and risk using entropy based confidence estimation with selective classifiers and assess its impact on the overall performance of Text-to-SQL models. Additionally, we explore the models' initial calibration and improve it with calibration techniques for better model alignment between confidence and accuracy. Our experimental results show that encoder-decoder T5 is better calibrated than in-context-learning GPT 4 and decoder-only Llama 3, thus the designated external entropy-based selective classifier has better performance. The study also reveal that, in terms of error detection, selective classifier with a higher probability detects errors associated with irrelevant questions rather than incorrect query generations.