mlsanityData to 14 Oct 2026

DirEAG: Dirichlet Evidence Aggregation for Calibrating Verbalized Confidence in Mathematical Reasoning

arXiv 2608.20717 ↗First posted 21 Aug 2026cs.AIcited by 0

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Reliable confidence estimation is essential for using large language models in mathematical reasoning, but black-box verbalized confidence is difficult to calibrate. When the same problem is queried under multiple confidence-steering prompts, the resulting answer-confidence observations contain useful uncertainty information, yet their scales may shift across steering levels, models, and datasets. Existing black-box uncertainty methods often rely on answer agreement, sample consistency, or entropy, which describe output variation but do not model the numerical meaning of self-reported confidence. Conversely, direct averaging or heuristic aggregation of elicited confidence cannot learn prompt- and task-dependent bias. We propose DirEAG, a Dirichlet Evidence Aggregation method that converts each elicited answer-confidence observation into calibrated soft evidence over generated candidate answers and an additional null state, allowing the model to represent cases where none of the candidates is correct. Experiments on GSM8K, SVAMP, and GSM-Hard with Qwen, Mistral, and Gemma models show that, compared with direct confidence averaging and heuristic confidence-steering aggregation, DirEAG often achieves better calibration while maintaining competitive answer selection. Ablations further reveal that evidence aggregation and final binary calibration address distinct parts of the calibration problem.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 18 of its 20 references that are in the map. First posted on arXiv 21 Aug 2026.

Builds on

18 of 20 references.

References · 18

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  4. 2017On Calibration of Modern Neural Networkscited by 4,960
  5. 2024Qwen2.5 Technical Reportcited by 4,942
  6. 2023Mistral 7Bcited by 3,164
  7. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  8. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  9. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 878
  10. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
  11. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 810
  12. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
6 more
  1. 2022PAL: Program-aided Language Modelscited by 652
  2. 2023Navigating the Grey Area: How Expressions of Uncertainty and Overconfidence Affect Language Modelscited by 91
  3. 2024Calibrating Large Language Models with Sample Consistencycited by 73
  4. 2025ConfTuner: Training Large Language Models to Express Their Confidence Verballycited by 39
  5. 2025SteerConf: Steering LLMs for Confidence Elicitationcited by 20
  6. 2025Calibrating Verbalized Confidence with Self-Generated Distractorscited by 14

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026How Uncertainty Estimation Scales with Sampling in Reasoning Models
  2. 2025CER: Confidence Enhanced Reasoning in LLMs
  3. 2026Probability is Not Enough: Exploring and Counting Divergent Tokens for Reasoning Uncertainty Quantification in LLMs
  4. 2026Post-Training Shifts Confidence: A Three-Stage Analysis of How SFT, RL, and OPD Shape CoT Calibration
  5. 2026Unsupervised Confidence Calibration for Reasoning LLMs from a Single Generation
  6. 2026Diagnosing Multi-step Reasoning Failures in Black-box LLMs via Stepwise Confidence Attribution
  7. 2026CALIBER: Calibrating Confidence Before and After Reasoning in Language Models
  8. 2025Optimizing Chain-of-Thought Confidence via Topological and Dirichlet Risk Analysis