mlsanityData to 7 Oct 2026

Calibrated Interpretation: Confidence Estimation in Semantic Parsing

arXiv 2211.07443 ↗First posted 14 Nov 2022cs.CLcited by 33

Not on the map. Why?

Abstract (arXiv)

Sequence generation models are increasingly being used to translate natural language into programs, i.e. to perform executable semantic parsing. The fact that semantic parsing aims to predict programs that can lead to executed actions in the real world motivates developing safe systems. This in turn makes measuring calibration -- a central component to safety -- particularly important. We investigate the calibration of popular generation models across four popular semantic parsing datasets, finding that it varies across models and datasets. We then analyze factors associated with calibration error and release new confidence-based challenge splits of two parsing datasets. To facilitate the inclusion of calibration in semantic parsing evaluations, we release a library for computing calibration metrics.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Builds on 58 of its 102 references that are in the map. First posted on arXiv 14 Nov 2022.

Builds on

58 of 102 references.

References · 58

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  9. 2017On Calibration of Modern Neural Networkscited by 4,860
  10. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  11. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  12. 2019The Curious Case of Neural Text Degenerationcited by 2,881
38 more
  1. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  2. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  3. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  4. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  5. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  6. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  7. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  8. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  9. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  10. 2021PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Modelscited by 402
  11. 2021Revisiting the Calibration of Modern Neural Networkscited by 387
  12. 2016Deep Biaffine Attention for Neural Dependency Parsingcited by 334
  13. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  14. 2020Calibration of Pre-trained Transformerscited by 307
  15. 2022Prompting GPT-3 To Be Reliablecited by 258
  16. 2022Interactive Language: Talking to Robots in Real Timecited by 217
  17. 2020Reducing conversational agents' overconfidence through linguistic calibrationcited by 209
  18. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 208
  19. 2019CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databasescited by 180
  20. 2021Constrained Language Models Yield Few-Shot Semantic Parserscited by 162
  21. 2020Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metricscited by 155
  22. 2020Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both?cited by 147
  23. 2020Semantic Evaluation for Text-to-SQL with Distilled Test Suitescited by 123
  24. 2020Task-Oriented Dialogue as Dataflow Synthesiscited by 122
  25. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 117
  26. 2018Semantic Parsing for Task Oriented Dialog using Hierarchical Representationscited by 101
  27. 2022Reliable Visual Question Answering: Abstain Rather Than Answer Incorrectlycited by 86
  28. 2022Inverse scaling can become U-shapedcited by 76
  29. 2019Calibration of Encoder Decoder Models for Neural Machine Translationcited by 67
  30. 2022Investigating Selective Prediction Approaches Across Several Tasks in IID, OOD, and Adversarial Settingscited by 66
  31. 2020On the Inference Calibration of Neural Machine Translationcited by 61
  32. 2019AMR Parsing as Sequence-to-Graph Transductioncited by 50
  33. 2021Few-Shot Semantic Parsing with Language Models Trained On Codecited by 50
  34. 2019Model-based Interactive Semantic Parsing: A Unified Framework and A Text-to-SQL Case Studycited by 40
  35. 2020Conversational Semantic Parsing for Dialog State Trackingcited by 40
  36. 2022Modern Baselines for SPARQL Semantic Parsingcited by 34
  37. 2018Confidence Modeling for Neural Semantic Parsingcited by 33
  38. 2019Broad-Coverage Semantic Parsing as Transductioncited by 32

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 9; 2025: 9; 2026: 8.

33 papers.

Sort:

By area:

  1. 2023ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMscited by 297
  2. 2024Don't Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaborationcited by 176
  3. 2024Language Model Cascades: Token-level uncertainty and beyondcited by 115
  4. 2025Uncertainty Quantification and Confidence Calibration in Large Language Models: A Surveycited by 93
  5. 2023Navigating the Grey Area: How Expressions of Uncertainty and Overconfidence Affect Language Modelscited by 91
  6. 2024BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Modelscited by 48
  7. 2023Zero and Few-shot Semantic Parsing with Ambiguous Inputscited by 16
  8. 2024TrustSQL: Benchmarking Text-to-SQL Reliability with Penalty-Based Scoringcited by 15
  9. 2025MICE for CATs: Model-Internal Confidence Estimation for Calibrating Agents with Toolscited by 15
  10. 2023Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Modelscited by 14
  11. 2024LACIE: Listener-Aware Finetuning for Confidence Calibration in Large Language Modelscited by 14
  12. 2024Teaching LLMs to Abstain across Languages via Multilingual Feedbackcited by 14
21 more
  1. 2025MIP against Agent: Malicious Image Patches Hijacking Multimodal OS Agentscited by 12
  2. 2023Simple Token-Level Confidence Improves Caption Correctnesscited by 11
  3. 2025Confidence Estimation for Error Detection in Text-to-SQL Systemscited by 10
  4. 2024Few-Shot Recalibration of Language Modelscited by 8
  5. 2024Text-to-SQL Calibration: No Need to Ask -- Just Rescale Model Probabilitiescited by 7
  6. 2023Did You Mean...? Confidence-based Trade-offs in Semantic Parsingcited by 5
  7. 2025GRACE: A Granular Benchmark for Evaluating Model Calibration against Human Calibrationcited by 5
  8. 2025Calibrating LLMs for Text-to-SQL Parsing by Leveraging Sub-clause Frequenciescited by 5
  9. 2025Generalized Correctness Models: Learning Calibrated and Model-Agnostic Correctness Predictors from Historical Patternscited by 5
  10. 2025Sample Smart, Not Hard: Correctness-First Decoding for Better Reasoning in LLMscited by 5
  11. 2026Calibration Is Not Enough: Evaluating Confidence Estimation Under Language Variationscited by 3
  12. 2025Confidence Estimation for Text-to-SQL in Large Language Modelscited by 2
  13. 2023Towards Unbiased Calibration using Meta-Regularizationcited by 1
  14. 2024Understanding the Effect of Algorithm Transparency of Model Explanations in Text-to-SQL Semantic Parsingcited by 1
  15. 2026The ACUTE Protocol: Operationalizing Language Model Activations for Better Calibration, Utility, and Trustcited by 1
  16. 2026SCP-NL2TL: Selective Conformal Prediction with Semantic Verification for Natural Language to Temporal Logic Specificationscited by 1
  17. 2026Calibrated Confidence Estimation for Tabular Question Answeringcited by 0
  18. 2026ECUASn: A family of metrics for principled evaluation of uncertainty-augmented systemscited by 0
  19. 2026Harnessing the Latent Space: From Steering Vectors to Model Calibrators for Control and Trustcited by 0
  20. 2026Relation Geometry in Semantic Space of Language Modelscited by 0
  21. 2026Certified Against Which Oracle? Execution Labels Set the Reported Risk of Conformal Abstention for Text-to-SQLcited by 0