Can Large Language Models Express Uncertainty Like Human?
Not on the map. Why?
Abstract (arXiv)
Large language models (LLMs) are increasingly used in high-stakes settings, where overconfident responses can mislead users. Reliable confidence estimation has been shown to enhance trust and task accuracy. Yet existing methods face practical barriers: logits are often hidden, multi-sampling is computationally expensive, and verbalized numerical uncertainty (e.g., giving a 0-100 score) deviates from natural communication. We revisit linguistic confidence (LC), where models express uncertainty through hedging language (e.g., probably, might), offering a lightweight and human-centered alternative. To advance this direction, we (1) release the first diverse, large-scale dataset of hedging expressions with human-annotated confidence scores, and (2) propose a lightweight mapper that converts hedges into confidence scores at near-zero cost. Building on these resources, we (3) conduct the first systematic study of LC across modern LLMs and QA benchmarks, revealing that while most LLMs underperform in expressing reliable LC, carefully designed prompting achieves competitive calibration and discriminability. Finally, we (4) introduce a fine-tuning framework that further improves LC reliability. Taken together, our work positions linguistic confidence as a scalable, efficient, and human-aligned approach to LLM uncertainty estimation, and calls for deeper exploration of this promising yet underexplored direction.
Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 22 of its 33 references that are in the map. First posted on arXiv 29 Sep 2025.
Builds on
22 of 33 references.
References · 22
- 2025Qwen3 Technical Reportcited by 8,820
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
- 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
- 2017On Calibration of Modern Neural Networkscited by 4,960
- 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
- 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
- 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
- 2022Language Models (Mostly) Know What They Knowcited by 1,856
- 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,128
- 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
- 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
- 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
10 more
- 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 695
- 2024Measuring short-form factuality in large language modelscited by 373
- 2024Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMscited by 259
- 2020Reducing conversational agents' overconfidence through linguistic calibrationcited by 210
- 2024Can Large Language Models Faithfully Express Their Intrinsic Uncertainty in Words?cited by 87
- 2024"I'm Not Sure, But...": Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trustcited by 67
- 2025Calibrating Verbal Uncertainty as a Linear Feature to Reduce Hallucinationscited by 48
- 2025From Calibration to Collaboration: LLM Uncertainty Quantification Should Be More Human-Centeredcited by 21
- 2024Perceptions of Linguistic Uncertainty by Language Models and Humanscited by 15
- 2024Calibrating Expressions of Certaintycited by 7
Built on by
Citing papers per year: 2026: 5.
5 papers.
- 2026Reinforcement Learning with Metacognitive Feedback Elicits Faithful Uncertainty Expression in LLMscited by 4
- 2026Confidence Before Answering: A Paradigm Shift for Efficient LLM Uncertainty Estimationcited by 3
- 2026Retrieval-Augmented Linguistic Calibrationcited by 2
- 2026Can LLMs Use Linguistic Uncertainty Markers to Reliably Reflect Intrinsic Confidence?cited by 2
- 2026Saying More Than They Know: A Framework for Quantifying Epistemic-Rhetorical Miscalibration in Large Language Modelscited by 0
- 2026Reinforcement Learning with Metacognitive Feedback Elicits Faithful Uncertainty Expression in LLMs30 Jun 2026
- 2026Can LLMs Use Linguistic Uncertainty Markers to Reliably Reflect Intrinsic Confidence?27 May 2026
- 2026Retrieval-Augmented Linguistic Calibration19 May 2026
- 2026Saying More Than They Know: A Framework for Quantifying Epistemic-Rhetorical Miscalibration in Large Language Models27 Mar 2026
- 2026Confidence Before Answering: A Paradigm Shift for Efficient LLM Uncertainty Estimation6 Mar 2026