mlsanityData to 14 Oct 2026

CALIBER: Calibrating Confidence Before and After Reasoning in Language Models

arXiv 2606.24281 ↗First posted 23 Jun 2026cs.CL cs.AIcited by 1

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Reasoning language models are increasingly asked not only to answer difficult questions, but also to estimate their likelihood of success. Existing methods typically elicit confidence only once: either before thinking or after answering. We argue that confidence in reasoning models is state-dependent: before thinking, confidence should estimate the chance of the model correctly solving the prompt, while after thinking it should predict whether the realized answer is likely to be correct. This distinction determines the appropriate supervision target: prompt-level success should supervise confidence estimates made after seeing the prompt, while individual answer-level correctness should supervise confidence estimates made after answering. We introduce CALIBER (Calibration Before and After Reasoning), which elicits both estimates and supervises each with the target matched to its information state. Under this unified protocol, CALIBER reduces Expected Calibration Error (ECE) by 52.5% over the strongest single-confidence baseline on BigMathDigits for the 7B model, while achieving the best Brier score and AUROC, and remains within 2.1 points of the best accuracy. Further, on a larger 30B model, CALIBER achieves the best ECE on BigMathDigits while remaining competitive in Brier score and AUROC. Out of distribution, it achieves the best ECE and Brier score on GPQA and TriviaQA, and remains competitive on SimpleQA. Ablations further show that this position-target alignment is most beneficial under distribution shift where it consistently reduces calibration error across all out-of-distribution benchmarks.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 27 of its 27 references that are in the map. First posted on arXiv 23 Jun 2026.

Builds on

27 of 27 references.

Closest ancestors · 2

Same direction, newest first

  1. 2026Confidence-Calibrated Small-Large Language Model Collaboration for Cost-Efficient Reasoning4 Mar
  2. 2022Language Models (Mostly) Know What They Know11 Jul

Other references · 25

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2017On Calibration of Modern Neural Networkscited by 4,960
  3. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  4. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  5. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,287
  6. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  7. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  8. 2017Selective Classification for Deep Neural Networkscited by 852
  9. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 819
  10. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
  11. 2024Measuring short-form factuality in large language modelscited by 373
  12. 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 256
13 more
  1. 2025Beyond Binary Rewards: Training LMs to Reason About Their Uncertaintycited by 117
  2. 2024Taming Overconfidence in LLMs: Reward Calibration in RLHFcited by 112
  3. 2025Big-Math: A Large-Scale, High-Quality Math Dataset for Reinforcement Learning in Language Modelscited by 109
  4. 2024SaySelf: Teaching LLMs to Express Confidence with Self-Reflective Rationalescited by 105
  5. 2024Large Language Models Must Be Taught to Know What They Don't Knowcited by 100
  6. 2025Command A: An Enterprise-Ready Large Language Modelcited by 59
  7. 2025Rewarding Doubt: A Reinforcement Learning Approach to Calibrated Confidence Expression of Large Language Modelscited by 52
  8. 2025Mind the Confidence Gap: Overconfidence, Calibration, and Distractor Effects in Large Language Modelscited by 40
  9. 2025ConfTuner: Training Large Language Models to Express Their Confidence Verballycited by 39
  10. 2026How do LLMs Compute Verbal Confidencecited by 26
  11. 2026Decoupling Reasoning and Confidence: Resurrecting Calibration in Reinforcement Learning from Verifiable Rewardscited by 14
  12. 2025Uncertainty Distillation: Teaching Language Models to Express Semantic Confidencecited by 13
  13. 2026Confidence Before Answering: A Paradigm Shift for Efficient LLM Uncertainty Estimationcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Test-time Calibration Learning for Large Language Model Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026ConfidenceBench: Evaluating Confidence Calibration in Large Language Models
  2. 2024Confidence in the Reasoning of Large Language Models
  3. 2025REHEARSE: Experiential Rehearsal for Verbal Confidence Calibration in Large Language Models
  4. 2026ORCE: Order-Aware Alignment of Verbalized Confidence in Large Language Models
  5. 2026The ACUTE Protocol: Operationalizing Language Model Activations for Better Calibration, Utility, and Trust
  6. 2026Post-Training Shifts Confidence: A Three-Stage Analysis of How SFT, RL, and OPD Shape CoT Calibration
  7. 2025Multiple Choice Questions: Reasoning Makes Large Language Models (LLMs) More Self-Confident, Especially When They are Wrong
  8. 2026LLMs Should Express Uncertainty Explicitly