mlsanityData to 7 Oct 2026

Calibrated Confidence Estimation for Tabular Question Answering

arXiv 2604.12491 ↗First posted 14 Apr 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) are increasingly deployed for tabular question answering, yet calibration on structured data is largely unstudied. This paper presents the first systematic comparison of five confidence estimation methods across five frontier LLMs and two tabular QA benchmarks. All models are severely overconfident (smooth ECE 0.35-0.64 versus 0.10-0.15 reported for textual QA). A consistent self-evaluation versus perturbation dichotomy replicates across both benchmarks and all four fully-covered models: self-evaluation methods (verbalized, P(True)) achieve AUROC 0.42-0.76, while perturbation methods (semantic entropy, self-consistency, and our Multi-Format Agreement) achieve AUROC 0.78-0.86. Per-model paired bootstrap tests reject the null at p<0.001 after Holm-Bonferroni correction, and a 3-seed check on GPT-4o-mini gives a per-seed standard deviation of only 0.006. The paper proposes Multi-Format Agreement (MFA), which exploits the lossless and deterministic serialization variation unique to structured data (Markdown, HTML, JSON, CSV) to estimate confidence at 20% lower API cost than sampling baselines. MFA reduces ECE by 44-63%, generalizes across all four models on TableBench (mean AUROC 0.80), and combines complementarily with sampling: an MFA + self-consistency ensemble lifts AUROC from 0.74 to 0.82. A secondary contribution, structure-aware recalibration, improves AUROC by +10 percentage points over standard post-hoc methods.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 38 of its 57 references that are in the map. First posted on arXiv 14 Apr 2026.

Builds on

38 of 57 references.

References · 38

  1. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  2. 2017On Calibration of Modern Neural Networkscited by 4,860
  3. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  4. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  5. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  6. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  7. 2017Selective Classification for Deep Neural Networkscited by 799
  8. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  9. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  10. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 667
  11. 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 552
  12. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 509
26 more
  1. 2019Verified Uncertainty Calibrationcited by 307
  2. 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 253
  3. 2024Chain-of-Table: Evolving Tables in the Reasoning Chain for Table Understandingcited by 185
  4. 2023Conformal Prediction with Large Language Models for Multi-Choice Question Answeringcited by 131
  5. 2024Language Models with Conformal Factuality Guaranteescited by 128
  6. 2024TableBench: A Comprehensive and Complex Benchmark for Table Question Answeringcited by 127
  7. 2024Benchmarking LLMs via Uncertainty Quantificationcited by 124
  8. 2023Table Meets LLM: Can Large Language Models Understand Structured Table Data? A Benchmark and Empirical Studycited by 116
  9. 2025Confidence Improves Self-Consistency in LLMscited by 111
  10. 2024Calibrating Large Language Models Using Their Generations Onlycited by 63
  11. 2023Smooth ECE: Principled Reliability Diagrams via Kernel Smoothingcited by 59
  12. 2024SPUQ: Perturbation-Based Uncertainty Quantification for Large Language Modelscited by 54
  13. 2023Tabular Representation, Noisy Operators, and Impacts on Table Structure Understanding Tasks in LLMscited by 49
  14. 2022Sample-dependent Adaptive Temperature Scaling for Improved Calibrationcited by 48
  15. 2024Elephants Never Forget: Memorization and Learning of Tabular Data in Large Language Modelscited by 34
  16. 2022Calibrated Interpretation: Confidence Estimation in Semantic Parsingcited by 33
  17. 2019Maximizing Overall Diversity for Improved Uncertainty Estimates in Deep Ensemblescited by 30
  18. 2025Semantic Energy: Detecting LLM Hallucination Beyond Entropycited by 18
  19. 2024Accurate and Regret-aware Numerical Problem Solver for Tabular Question Answeringcited by 10
  20. 2024Few-Shot Recalibration of Language Modelscited by 8
  21. 2024Text-to-SQL Calibration: No Need to Ask -- Just Rescale Model Probabilitiescited by 7
  22. 2026On Calibration of Large Language Models: From Response To Capabilitycited by 5
  23. 2025STaR: Towards Effective and Stable Table Reasoning via Slow-Thinking Large Language Modelscited by 3
  24. 2026Semantic Calibration Prevails Where Token Confidence Fails: Benchmarking Long-Form Scientific QAcited by 3
  25. 2025Exploring Generative Process Reward Modeling for Semi-Structured Data: A Case Study of Table Question Answeringcited by 2
  26. 2026Self-Improving Code Generation via Semantic Entropy and Behavioral Consensuscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026A Semantic-Sampling Framework for Evaluating Calibration in Open-Ended Question Answering
  2. 2026ConfidenceBench: Evaluating Confidence Calibration in Large Language Models
  3. 2023On the Calibration of Multilingual Question Answering LLMs
  4. 2024Calibrating the Confidence of Large Language Models by Eliciting Fidelity
  5. 2026Uncertainty-Aware Abstention in Large Language Models with Provable Alignment Guarantees
  6. 2026ORCE: Order-Aware Alignment of Verbalized Confidence in Large Language Models
  7. 2024MAQA: Evaluating Uncertainty Quantification in LLMs Regarding Data Uncertainty
  8. 2025Trained on Tokens, Calibrated on Concepts: The Emergence of Semantic Calibration in LLMs