mlsanityData to 14 Oct 2026

Can Large Language Models Express Uncertainty Like Human?

arXiv 2509.24202 ↗First posted 29 Sep 2025cs.CL cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) are increasingly used in high-stakes settings, where overconfident responses can mislead users. Reliable confidence estimation has been shown to enhance trust and task accuracy. Yet existing methods face practical barriers: logits are often hidden, multi-sampling is computationally expensive, and verbalized numerical uncertainty (e.g., giving a 0-100 score) deviates from natural communication. We revisit linguistic confidence (LC), where models express uncertainty through hedging language (e.g., probably, might), offering a lightweight and human-centered alternative. To advance this direction, we (1) release the first diverse, large-scale dataset of hedging expressions with human-annotated confidence scores, and (2) propose a lightweight mapper that converts hedges into confidence scores at near-zero cost. Building on these resources, we (3) conduct the first systematic study of LC across modern LLMs and QA benchmarks, revealing that while most LLMs underperform in expressing reliable LC, carefully designed prompting achieves competitive calibration and discriminability. Finally, we (4) introduce a fine-tuning framework that further improves LC reliability. Taken together, our work positions linguistic confidence as a scalable, efficient, and human-aligned approach to LLM uncertainty estimation, and calls for deeper exploration of this promising yet underexplored direction.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 22 of its 33 references that are in the map. First posted on arXiv 29 Sep 2025.

Builds on

22 of 33 references.

References · 22

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  4. 2017On Calibration of Modern Neural Networkscited by 4,960
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  6. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
  7. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  8. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  9. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,128
  10. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  11. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
  12. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
10 more
  1. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 695
  2. 2024Measuring short-form factuality in large language modelscited by 373
  3. 2024Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMscited by 259
  4. 2020Reducing conversational agents' overconfidence through linguistic calibrationcited by 210
  5. 2024Can Large Language Models Faithfully Express Their Intrinsic Uncertainty in Words?cited by 87
  6. 2024"I'm Not Sure, But...": Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trustcited by 67
  7. 2025Calibrating Verbal Uncertainty as a Linear Feature to Reduce Hallucinationscited by 48
  8. 2025From Calibration to Collaboration: LLM Uncertainty Quantification Should Be More Human-Centeredcited by 21
  9. 2024Perceptions of Linguistic Uncertainty by Language Models and Humanscited by 15
  10. 2024Calibrating Expressions of Certaintycited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.