mlsanityData to 14 Oct 2026

Certified Against Which Oracle? Execution Labels Set the Reported Risk of Conformal Abstention for Text-to-SQL

arXiv 2609.25938 ↗First posted 22 Sep 2026cs.LG cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

A conformal abstention certificate for text-to-SQL is only as truthful as the correctness labels it is calibrated on. The uncertainty pipelines that read confidence off execution consistency take those labels from the single database a benchmark ships, an oracle known to be lenient. We run a preregistered intervention on Spider-Realistic, swapping that database for the benchmark's distilled multi-instance test suite. Across four SQL-specialist checkpoints and two split schemes, the swap raises the certificate's held-out risk 2.73 to 10.23 points above the risk its own labels report. Neither oracle reports the risk experts assign. Under blinded labels from two SQL experts, a certificate calibrated at a nominal 0.10 carries 20.0 and 17.2 points of risk on two checkpoints. The stricter oracle errs in both directions: most of the answers it rejects are not judged wrong, and some of those it accepts are. An AI-assigned census of what it rejects finds a semantic error in a quarter to a third of them, depending on the population. It attributes most of the rest to underspecified questions, synthetic instances or suspected reference-query defects, a flag supported by a preregistered blinded expert audit. The oracle also decides how a confidence score is judged. Every execution-consistency score looks better under the labels of the oracle that built its clusters, in 16 of 16 combinations. Under expert labels, building such a score on suite clusters instead of shipped-database clusters raises its area under the ROC curve (AUROC) by 6.96 points on one checkpoint and 1.53 on the other. On the second, the expert interval excludes the 8.3 points the suite labels report. A certificate should be reported with both oracles, and an oracle-relative difference read as semantic risk only after the benchmark is audited. A consistency score should be evaluated under an oracle that did not build it.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 48 of its 56 references that are in the map. First posted on arXiv 22 Sep 2026.

Builds on

48 of 56 references.

References · 48

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  3. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,139
  4. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  5. 2017Selective Classification for Deep Neural Networkscited by 852
  6. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 819
  7. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 688
  8. 2021Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarkscited by 395
  9. 2022Conformal Risk Controlcited by 299
  10. 2023Generating with Confidence: Uncertainty Quantification for Black-box Large Language Modelscited by 293
  11. 2021Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Controlcited by 271
  12. 2023Conformal Language Modelingcited by 170
36 more
  1. 2023LM-Polygraph: Uncertainty Estimation for Language Modelscited by 131
  2. 2024Language Models with Conformal Factuality Guaranteescited by 131
  3. 2020Semantic Evaluation for Text-to-SQL with Distilled Test Suitescited by 126
  4. 2020Structure-Grounded Pretraining for Text-to-SQLcited by 126
  5. 2025OmniSQL: Synthesizing High-quality Text-to-SQL Data at Scalecited by 91
  6. 2025XiYan-SQL: A Novel Multi-Generator Framework For Text-to-SQLcited by 40
  7. 2024Understanding the Effects of Noise in Text-to-SQL: An Examination of the BIRD-Bench Benchmarkcited by 34
  8. 2022Calibrated Interpretation: Confidence Estimation in Semantic Parsingcited by 33
  9. 2023Evaluating Cross-Domain Text-to-SQL Models and Benchmarkscited by 32
  10. 2023Adaptive conformal classification with noisy labelscited by 25
  11. 2022Label Noise Robustness of Conformal Predictioncited by 20
  12. 2025Reliable Text-to-SQL with Adaptive Abstentioncited by 20
  13. 2025Uncertainty Quantification for Language Models: A Suite of Black-Box, White-Box, LLM Judge, and Ensemble Scorerscited by 18
  14. 2024TrustSQL: Benchmarking Text-to-SQL Reliability with Penalty-Based Scoringcited by 17
  15. 2023Universal distribution of the empirical coverage in split conformal predictioncited by 15
  16. 2026Pervasive Annotation Errors Break Text-to-SQL Benchmarks and Leaderboardscited by 13
  17. 2023Error Detection for Text-to-SQL Semantic Parsingcited by 12
  18. 2024FLEX: Expert-level False-Less EXecution Metric for Reliable Text-to-SQL Benchmarkcited by 11
  19. 2025Confidence Estimation for Error Detection in Text-to-SQL Systemscited by 11
  20. 2024Text-to-SQL Calibration: No Need to Ask -- Just Rescale Model Probabilitiescited by 7
  21. 2025Addressing Pitfalls in the Evaluation of Uncertainty Estimation Methods for Natural Language Generationcited by 7
  22. 2025Calibrating LLMs for Text-to-SQL Parsing by Leveraging Sub-clause Frequenciescited by 5
  23. 2026Geometry-Calibrated Conformal Abstention for Language Modelscited by 5
  24. 2025SpotIt: Evaluating Text-to-SQL Evaluation with Formal Verificationcited by 4
  25. 2026DPC: Training-Free Text-to-SQL Candidate Selection via Dual-Paradigm Consistencycited by 4
  26. 2026When Can Conformal Risk Control Certify LLM Outputs? Bounds, Impossibility, and Adaptation for Structured Generationcited by 4
  27. 2024ETM: Modern Insights into Perspective on Text-to-SQL Evaluation in the Age of Large Language Modelscited by 3
  28. 2025Confidence Estimation for Text-to-SQL in Large Language Modelscited by 3
  29. 2026Disentangling Ambiguity from Instability in Large Language Models: A Clinical Text-to-SQL Case Studycited by 3
  30. 2026Functional Entropy: Predicting Functional Correctness in LLM-Generated Code with Uncertainty Quantificationcited by 3
  31. 2026SynSQL: Synthesizing Relational Databases for Robust Evaluation of Text-to-SQL Systemscited by 2
  32. 2026Test-Time Verification for Text-to-SQL via Outcome Reward Modelscited by 2
  33. 2026What Predicts Correctness in Text-to-SQL? A Selective-Prediction Studycited by 2
  34. 2026SpotIt+: Verification-based Text-to-SQL Evaluation with Database Constraintscited by 1
  35. 2026Knowing When to Stop: Predicting Execution-Consistency Convergence in Text-to-SQLcited by 1
  36. 2026Never the Number: Structural Abstention for AI Systems Whose Answers Are Consumed as Factcited by 1

Built on by

Not cited yet.