# LLMs Should Express Uncertainty Explicitly arXiv: https://arxiv.org/abs/2604.05306 ยท HTML: https://mlsanity.com/paper/2604.05306/ ## Key facts - Cited by 6 papers in the map (data through 14 Oct 2026). - Builds on 50 of its 54 references that are in the map. - First posted on arXiv 7 Apr 2026. ## Builds on (most cited first, 30 of 50) - [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022) - [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024) - [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017) - [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018) - [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017) - [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024) - [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022) - [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://mlsanity.com/paper/2310.11511/) (2023) - [Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps](https://mlsanity.com/paper/2011.01060/) (2020) - [Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs](https://mlsanity.com/paper/2306.13063/) (2023) - [MuSiQue: Multihop Questions via Single-hop Question Composition](https://mlsanity.com/paper/2108.00573/) (2021) - [Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation](https://mlsanity.com/paper/2302.09664/) (2023) - [Discovering Latent Knowledge in Language Models Without Supervision](https://mlsanity.com/paper/2212.03827/) (2022) - [Teaching Models to Express Their Uncertainty in Words](https://mlsanity.com/paper/2205.14334/) (2022) - [The Internal State of an LLM Knows When It's Lying](https://mlsanity.com/paper/2304.13734/) (2023) - [Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning](https://mlsanity.com/paper/2506.01939/) (2025) - [Active Retrieval Augmented Generation](https://mlsanity.com/paper/2305.06983/) (2023) - [Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity](https://mlsanity.com/paper/2403.14403/) (2024) - [Learning to Compress Prompts with Gist Tokens](https://mlsanity.com/paper/2304.08467/) (2023) - [Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models](https://mlsanity.com/paper/2305.19187/) (2023) - [Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMs](https://mlsanity.com/paper/2406.15927/) (2024) - [Learning to Reason without External Rewards](https://mlsanity.com/paper/2505.19590/) (2025) - [Deep Think with Confidence](https://mlsanity.com/paper/2508.15260/) (2025) - [Taming Overconfidence in LLMs: Reward Calibration in RLHF](https://mlsanity.com/paper/2410.09724/) (2024) - [Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph](https://mlsanity.com/paper/2406.15627/) (2024) - [Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity](https://mlsanity.com/paper/2510.01171/) (2025) - [Large Language Models Must Be Taught to Know What They Don't Know](https://mlsanity.com/paper/2406.08391/) (2024) - [Can Large Language Models Faithfully Express Their Intrinsic Uncertainty in Words?](https://mlsanity.com/paper/2405.16908/) (2024) - [On Verbalized Confidence Scores for LLMs](https://mlsanity.com/paper/2412.14737/) (2024) - [LLM Internal States Reveal Hallucination Risk Faced With a Query](https://mlsanity.com/paper/2407.03282/) (2024) ## Built on by (most cited first, 6 of 6) - [From Model Scaling to System Scaling: Scaling the Harness in Agentic AI](https://mlsanity.com/paper/2605.26112/) (2026) - [Quantifying Faithful Confidence Expression in Large Reasoning Models](https://mlsanity.com/paper/2606.03969/) (2026) - [Know When to Trust the Skill: Delayed Appraisal and Epistemic Vigilance for Single-Agent LLMs](https://mlsanity.com/paper/2604.16753/) (2026) - [Calibration, Not Answer Selection: Distilling Internal Confidence in Reasoning Models](https://mlsanity.com/paper/2609.33290/) (2026) - [When Words Fall Short: Iterative Synergy Between Verbalized Reasoning and Hidden Features for LLM Confidence Estimation](https://mlsanity.com/paper/2609.34454/) (2026) - [Nudgeability: Reasoning Models Follow Confidence Signals Without Tracking Their Own Competence](https://mlsanity.com/paper/2609.34572/) (2026) ## Abstract (arXiv) Large language models (LLMs) often produce confident yet incorrect answers, which can lead to risky failures in real-world applications. We study whether post-training can make a model's self-assessment explicit: when the model is uncertain, can it be trained to signal so within its own response? A central design question is where in the response this signal should be exposed -- during reasoning, while the answer is still being formed, or at the end, once the answer has been produced. We study both. For end-of-reasoning self-assessment, we train the model to verbalize a confidence score for its response, with the aim of high confidence on correct answers and low confidence on incorrect ones. For during-reasoning self-assessment, we train the model to emit the marker whenever its current reasoning state appears unreliable. Across factual reasoning tasks, both forms sharply reduce overconfident errors while improving answer quality, and both can be used as triggers for retrieval augmented generation (RAG) to improve the final response. We further analyze their internal mechanisms: end-of-reasoning verbalized confidence sharpens a confidence-related structure already present in the pretrained model, whereas during-reasoning emission teaches the model to mark high-risk reasoning steps, with parameter changes concentrated in the model's late layers.