# Are LLM Decisions Faithful to Verbal Confidence?

arXiv: https://arxiv.org/abs/2601.07767 · HTML: https://mlsanity.com/paper/2601.07767/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 36 references that are in the map.
- First posted on arXiv 12 Jan 2026.

## Builds on (most cited first, 20 of 20)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs](https://mlsanity.com/paper/2306.13063/) (2023)
- [Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback](https://mlsanity.com/paper/2305.14975/) (2023)
- [Teaching Models to Express Their Uncertainty in Words](https://mlsanity.com/paper/2205.14334/) (2022)
- [A Survey of Confidence Estimation and Calibration in Large Language Models](https://mlsanity.com/paper/2311.08298/) (2023)
- [AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions](https://mlsanity.com/paper/2506.09038/) (2025)
- [Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty](https://mlsanity.com/paper/2507.16806/) (2025)
- [DisentQA: Disentangling Parametric and Contextual Knowledge with Counterfactual Question Answering](https://mlsanity.com/paper/2211.05655/) (2022)
- [Reasoning Models Better Express Their Confidence](https://mlsanity.com/paper/2505.14489/) (2025)
- [Calibrating the Confidence of Large Language Models by Eliciting Fidelity](https://mlsanity.com/paper/2404.02655/) (2024)
- [LLM economicus? Mapping the Behavioral Biases of LLMs via Utility Theory](https://mlsanity.com/paper/2408.02784/) (2024)
- [Decision-Making Behavior Evaluation Framework for LLMs under Uncertain Context](https://mlsanity.com/paper/2406.05972/) (2024)
- [Atomic Calibration of LLMs in Long-Form Generations](https://mlsanity.com/paper/2410.13246/) (2024)
- [From Calibration to Collaboration: LLM Uncertainty Quantification Should Be More Human-Centered](https://mlsanity.com/paper/2506.07461/) (2025)
- [Trace Length is a Simple Uncertainty Signal in Reasoning Models](https://mlsanity.com/paper/2510.10409/) (2025)
- [LoVeC: Reinforcement Learning for Better Verbalized Confidence in Long-Form Generations](https://mlsanity.com/paper/2505.23912/) (2025)
- [Teaching LLMs to Abstain across Languages via Multilingual Feedback](https://mlsanity.com/paper/2406.15948/) (2024)
- [Evaluating and Aligning Human Economic Risk Preferences in LLMs](https://mlsanity.com/paper/2503.06646/) (2025)

## Built on by (most cited first, 8 of 8)

- [I-CALM: Incentivizing Confidence-Aware Abstention for LLM Selective Answering](https://mlsanity.com/paper/2604.03904/) (2026)
- [CoT2-Meta: Budgeted Metacognitive Control for Test-Time Reasoning](https://mlsanity.com/paper/2603.28135/) (2026)
- [Reflective Prompt Tuning through Language Model Function-Calling](https://mlsanity.com/paper/2605.21781/) (2026)
- [When Policies Change Probabilities: Modular Decision-Making for LLM Code Review](https://mlsanity.com/paper/2608.02677/) (2026)
- [Pander Score: A Continuous Measure of Sycophancy as Epistemic Deference](https://mlsanity.com/paper/2606.07897/) (2026)
- [Stable Miscalibration in Large Language Models: A Practical View of High-Confidence Errors](https://mlsanity.com/paper/2608.13591/) (2026)
- [Penalty-Framed No-Valid-Option MCQA: Analyzing LLM Abstention under Invalid Choices](https://mlsanity.com/paper/2610.08153/) (2026)
- [A Tale of Two Error Categories: Exploring Concealed Trade-Offs in the Errors of Automated Judges in Evaluation of Uncertainty Quantifiers](https://mlsanity.com/paper/2610.09693/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) can produce surprisingly sophisticated estimates of their own uncertainty. However, it remains unclear to what extent this expressed confidence is tied to the reasoning, knowledge, or decision making of the model. To test this, we introduce RiskEval: a framework designed to evaluate whether models adjust their abstention policies in response to varying error penalties. Our evaluation of several frontier models reveals a critical dissociation: models are neither cost-aware when articulating their verbal confidence, nor strategically responsive when deciding whether to engage or abstain under high-penalty conditions. Even when extreme penalties render frequent abstention the mathematically optimal strategy, models almost never abstain, resulting in utility collapse. This indicates that calibrated verbal confidence scores may not be sufficient to create trustworthy and interpretable AI systems, as current models lack the strategic agency to convert uncertainty signals into optimal and risk-sensitive decisions.