mlsanityData to 7 Oct 2026

Did You Mean...? Confidence-based Trade-offs in Semantic Parsing

arXiv 2303.16857 ↗First posted 29 Mar 2023cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

We illustrate how a calibrated model can help balance common trade-offs in task-oriented parsing. In a simulated annotator-in-the-loop experiment, we show that well-calibrated confidence scores allow us to balance cost with annotator load, improving accuracy with a small number of interactions. We then examine how confidence scores can help optimize the trade-off between usability and safety. We show that confidence-based thresholding can substantially reduce the number of incorrect low-confidence programs executed; however, this comes at a cost to usability. We propose the DidYouMean system which better balances usability and safety.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 18 of its 36 references that are in the map. First posted on arXiv 29 Mar 2023.

Builds on

18 of 36 references.

References · 18

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  2. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  3. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  4. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  5. 2022Interactive Language: Talking to Robots in Real Timecited by 217
  6. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 208
  7. 2020Task-Oriented Dialogue as Dataflow Synthesiscited by 122
  8. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 117
  9. 2018Semantic Parsing for Task Oriented Dialog using Hierarchical Representationscited by 101
  10. 2022Reliable Visual Question Answering: Abstain Rather Than Answer Incorrectlycited by 86
  11. 2022Investigating Selective Prediction Approaches Across Several Tasks in IID, OOD, and Adversarial Settingscited by 66
  12. 2019AMR Parsing as Sequence-to-Graph Transductioncited by 50
6 more
  1. 2019Model-based Interactive Semantic Parsing: A Unified Framework and A Text-to-SQL Case Studycited by 40
  2. 2020Conversational Semantic Parsing for Dialog State Trackingcited by 40
  3. 2022Calibrated Interpretation: Confidence Estimation in Semantic Parsingcited by 33
  4. 2019Broad-Coverage Semantic Parsing as Transductioncited by 32
  5. 2022The Whole Truth and Nothing But the Truth: Faithful and Controllable Dialogue Response Generation with Dataflow Transduction and Constrained Decodingcited by 7
  6. 2022When More Data Hurts: A Troubling Quirk in Developing Broad-Coverage Natural Language Understanding Systemscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 0; 2025: 2; 2026: 2.

5 papers.

  1. 2023Zero and Few-shot Semantic Parsing with Ambiguous Inputscited by 16
  2. 2025MICE for CATs: Model-Internal Confidence Estimation for Calibrating Agents with Toolscited by 15
  3. 2025Generalized Correctness Models: Learning Calibrated and Model-Agnostic Correctness Predictors from Historical Patternscited by 5
  4. 2026Completing Missing Annotation: Multi-Agent Debate for Accurate and Scalable Relevant Assessment for IR Benchmarkscited by 1
  5. 2026The ACUTE Protocol: Operationalizing Language Model Activations for Better Calibration, Utility, and Trustcited by 1