mlsanityData to 14 Oct 2026

An Evaluation of Estimative Uncertainty in Large Language Models

arXiv 2405.15185 ↗First posted 24 May 2024cs.CL cs.AI cs.HCcited by 13

Not on the map. Why?

Abstract (arXiv)

Words of estimative probability (WEPs), such as ''maybe'' or ''probably not'' are ubiquitous in natural language for communicating estimative uncertainty, compared with direct statements involving numerical probability. Human estimative uncertainty, and its calibration with numerical estimates, has long been an area of study -- including by intelligence agencies like the CIA. This study compares estimative uncertainty in commonly used large language models (LLMs) like GPT-4 and ERNIE-4 to that of humans, and to each other. Here we show that LLMs like GPT-3.5 and GPT-4 align with human estimates for some, but not all, WEPs presented in English. Divergence is also observed when the LLM is presented with gendered roles and Chinese contexts. Further study shows that an advanced LLM like GPT-4 can consistently map between statistical and estimative uncertainty, but a significant performance gap remains. The results contribute to a growing body of research on human-LLM alignment.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 27 of its 94 references that are in the map. First posted on arXiv 24 May 2024.

Builds on

27 of 94 references.

References · 27

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  8. 2023A Survey of Large Language Modelscited by 2,576
  9. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 923
  10. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  11. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 659
  12. 2020Artificial Intelligence, Values and Alignmentcited by 448
15 more
  1. 2023Benchmarking Large Language Models for News Summarizationcited by 445
  2. 2022Using cognitive psychology to understand GPT-3cited by 398
  3. 2023AI Alignment: A Comprehensive Surveycited by 263
  4. 2023Prompting Large Language Model for Machine Translation: A Case Studycited by 250
  5. 2023Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarizationcited by 246
  6. 2018Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systemscited by 234
  7. 2023Gender bias and stereotypes in Large Language Modelscited by 187
  8. 2023Cultural Bias and Cultural Alignment of Large Language Modelscited by 149
  9. 2024What Large Language Models Know and What People Think They Knowcited by 110
  10. 2022Putting GPT-3's Creativity to the (Alternative Uses) Testcited by 62
  11. 2021Shared Interest: Measuring Human-AI Alignment to Identify Recurring Patterns in Model Behaviorcited by 29
  12. 2023Inductive reasoning in humans and large language modelscited by 24
  13. 2024ACLSum: A New Dataset for Aspect-based Summarization of Scientific Publicationscited by 20
  14. 2023Evaluating the Deductive Competence of Large Language Modelscited by 12
  15. 2022Probing neural language models for understanding of words of estimative probabilitycited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 6; 2026: 6.

13 papers.

Sort:

By area:

  1. 2025Revisiting Epistemic Markers in Confidence Estimation: Can Markers Accurately Reflect Large Language Models' Uncertainty?cited by 16
  2. 2025MetaFaith: Faithful Natural Language Uncertainty Expression in LLMscited by 16
  3. 2024Perceptions of Linguistic Uncertainty by Language Models and Humanscited by 15
  4. 2025Anthropomimetic Uncertainty: What Verbalized Uncertainty in Language Models is Missingcited by 14
  5. 2026Agent-BRACE: Decoupling Beliefs from Actions in Long-Horizon Tasks via Verbalized State Uncertaintycited by 10
  6. 2026Reinforcement Learning with Metacognitive Feedback Elicits Faithful Uncertainty Expression in LLMscited by 4
  7. 2025Diagnosing Failures in Large Language Models' Answers: Integrating Error Attribution into Evaluation Frameworkcited by 3
  8. 2025Investigating Human-Aligned Large Language Model Uncertaintycited by 2
  9. 2025ChemAU: Harness the Reasoning of LLMs in Chemical Research with Adaptive Uncertainty Estimationcited by 2
  10. 2026Can LLMs Use Linguistic Uncertainty Markers to Reliably Reflect Intrinsic Confidence?cited by 2
  11. 2026Does Slightly Mean Somewhat? Measuring Vague Intensity Words in LLM Numeric Actionscited by 0
  12. 2026Measuring Behavioural Signatures of Large Language Models through Psychometric Profilingcited by 0
1 more
  1. 2026Bayesian Fine-tuning Yields Language Models that are as Bayesian as their Beliefs Allowcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Human-Alignment and Calibration of Inference-Time Uncertainty in Large Language Models
  2. 2026Rethinking Uncertainty Evaluation in Large Language Models
  3. 2024Rethinking the Uncertainty: A Critical Review and Analysis in the Era of Large Language Models
  4. 2026Human-Alignment, Calibration, and Activation Patterns in Large Language Model Uncertainty
  5. 2025Revisiting Uncertainty Estimation and Calibration of Large Language Models
  6. 2025OpenEstimate: Evaluating LLMs on Reasoning Under Uncertainty with Real-World Data
  7. 2024Confidence Under the Hood: An Investigation into the Confidence-Probability Alignment in Large Language Models
  8. 2026Estimating Uncertainty from Reasoning: A Large-Scale Study of Multi- and Crosslingual MCQA Performance in LLMs