# Calibrated Confidence Estimation for Tabular Question Answering

arXiv: https://arxiv.org/abs/2604.12491 · HTML: https://mlsanity.com/paper/2604.12491/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 38 of its 57 references that are in the map.
- First posted on arXiv 14 Apr 2026.

## Builds on (most cited first, 30 of 38)

- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [Large Language Models Cannot Self-Correct Reasoning Yet](https://mlsanity.com/paper/2310.01798/) (2023)
- [Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs](https://mlsanity.com/paper/2306.13063/) (2023)
- [Selective Classification for Deep Neural Networks](https://mlsanity.com/paper/1705.08500/) (2017)
- [Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation](https://mlsanity.com/paper/2302.09664/) (2023)
- [Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback](https://mlsanity.com/paper/2305.14975/) (2023)
- [Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs](https://mlsanity.com/paper/2305.03111/) (2023)
- [FinQA: A Dataset of Numerical Reasoning over Financial Data](https://mlsanity.com/paper/2109.00122/) (2021)
- [Compositional Semantic Parsing on Semi-Structured Tables](https://mlsanity.com/paper/1508.00305/) (2015)
- [Verified Uncertainty Calibration](https://mlsanity.com/paper/1909.10155/) (2019)
- [A Survey of Confidence Estimation and Calibration in Large Language Models](https://mlsanity.com/paper/2311.08298/) (2023)
- [Chain-of-Table: Evolving Tables in the Reasoning Chain for Table Understanding](https://mlsanity.com/paper/2401.04398/) (2024)
- [Conformal Prediction with Large Language Models for Multi-Choice Question Answering](https://mlsanity.com/paper/2305.18404/) (2023)
- [Language Models with Conformal Factuality Guarantees](https://mlsanity.com/paper/2402.10978/) (2024)
- [TableBench: A Comprehensive and Complex Benchmark for Table Question Answering](https://mlsanity.com/paper/2408.09174/) (2024)
- [Benchmarking LLMs via Uncertainty Quantification](https://mlsanity.com/paper/2401.12794/) (2024)
- [Table Meets LLM: Can Large Language Models Understand Structured Table Data? A Benchmark and Empirical Study](https://mlsanity.com/paper/2305.13062/) (2023)
- [Confidence Improves Self-Consistency in LLMs](https://mlsanity.com/paper/2502.06233/) (2025)
- [Calibrating Large Language Models Using Their Generations Only](https://mlsanity.com/paper/2403.05973/) (2024)
- [Smooth ECE: Principled Reliability Diagrams via Kernel Smoothing](https://mlsanity.com/paper/2309.12236/) (2023)
- [SPUQ: Perturbation-Based Uncertainty Quantification for Large Language Models](https://mlsanity.com/paper/2403.02509/) (2024)
- [Tabular Representation, Noisy Operators, and Impacts on Table Structure Understanding Tasks in LLMs](https://mlsanity.com/paper/2310.10358/) (2023)
- [Sample-dependent Adaptive Temperature Scaling for Improved Calibration](https://mlsanity.com/paper/2207.06211/) (2022)
- [Elephants Never Forget: Memorization and Learning of Tabular Data in Large Language Models](https://mlsanity.com/paper/2404.06209/) (2024)
- [Calibrated Interpretation: Confidence Estimation in Semantic Parsing](https://mlsanity.com/paper/2211.07443/) (2022)
- [Maximizing Overall Diversity for Improved Uncertainty Estimates in Deep Ensembles](https://mlsanity.com/paper/1906.07380/) (2019)
- [Semantic Energy: Detecting LLM Hallucination Beyond Entropy](https://mlsanity.com/paper/2508.14496/) (2025)

## Abstract (arXiv)

Large language models (LLMs) are increasingly deployed for tabular question answering, yet calibration on structured data is largely unstudied. This paper presents the first systematic comparison of five confidence estimation methods across five frontier LLMs and two tabular QA benchmarks. All models are severely overconfident (smooth ECE 0.35-0.64 versus 0.10-0.15 reported for textual QA). A consistent self-evaluation versus perturbation dichotomy replicates across both benchmarks and all four fully-covered models: self-evaluation methods (verbalized, P(True)) achieve AUROC 0.42-0.76, while perturbation methods (semantic entropy, self-consistency, and our Multi-Format Agreement) achieve AUROC 0.78-0.86. Per-model paired bootstrap tests reject the null at p<0.001 after Holm-Bonferroni correction, and a 3-seed check on GPT-4o-mini gives a per-seed standard deviation of only 0.006. The paper proposes Multi-Format Agreement (MFA), which exploits the lossless and deterministic serialization variation unique to structured data (Markdown, HTML, JSON, CSV) to estimate confidence at 20% lower API cost than sampling baselines. MFA reduces ECE by 44-63%, generalizes across all four models on TableBench (mean AUROC 0.80), and combines complementarily with sampling: an MFA + self-consistency ensemble lifts AUROC from 0.74 to 0.82. A secondary contribution, structure-aware recalibration, improves AUROC by +10 percentage points over standard post-hoc methods.