mlsanityData to 14 Oct 2026

Perceptions of Linguistic Uncertainty by Language Models and Humans

arXiv 2407.15814 ↗First posted 22 Jul 2024cs.CL cs.AI cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

_Uncertainty expressions_ such as "probably" or "highly unlikely" are pervasive in human language. While prior work has established that there is population-level agreement in terms of how humans quantitatively interpret these expressions, there has been little inquiry into the abilities of language models in the same context. In this paper, we investigate how language models map linguistic expressions of uncertainty to numerical responses. Our approach assesses whether language models can employ theory of mind in this setting: understanding the uncertainty of another agent about a particular statement, independently of the model's own certainty about that statement. We find that 7 out of 10 models are able to map uncertainty expressions to probabilistic responses in a human-like manner. However, we observe systematically different behavior depending on whether a statement is actually true or false. This sensitivity indicates that language models are substantially more susceptible to bias based on their prior knowledge (as compared to humans). These findings raise important questions and have broad implications for human-AI and AI-AI communication.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 50 of its 101 references that are in the map. First posted on arXiv 22 Jul 2024.

Builds on

50 of 101 references.

References · 50

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  5. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  7. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  8. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  9. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  10. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  11. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  12. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 819
38 more
  1. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
  2. 2023Whose Opinions Do Language Models Reflect?cited by 773
  3. 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 752
  4. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
  5. 2022Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studiescited by 515
  6. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 445
  7. 2021Entity-Based Knowledge Conflicts in Question Answeringcited by 356
  8. 2023Generating with Confidence: Uncertainty Quantification for Black-box Large Language Modelscited by 293
  9. 2024Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?cited by 232
  10. 2022Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMscited by 220
  11. 2023Evaluating the Moral Beliefs Encoded in LLMscited by 215
  12. 2020Reducing conversational agents' overconfidence through linguistic calibrationcited by 210
  13. 2023"Kelly is a Warm Person, Joseph is a Role Model": Gender Biases in LLM-Generated Reference Letterscited by 199
  14. 2023Gender bias and stereotypes in Large Language Modelscited by 187
  15. 2023A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicitycited by 182
  16. 2023In-Context Impersonation Reveals Large Language Models' Strengths and Biasescited by 155
  17. 2024Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Modelscited by 154
  18. 2023Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Modelscited by 150
  19. 2023Quantifying Uncertainty in Answers from any Language Model and Enhancing their Trustworthinesscited by 122
  20. 2024Tokenization counts: the impact of tokenization on arithmetic in frontier LLMscited by 115
  21. 2023Decomposing Uncertainty for Large Language Models through Input Clarification Ensemblingcited by 114
  22. 2024What Large Language Models Know and What People Think They Knowcited by 110
  23. 2024Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertaintycited by 106
  24. 2024Can Large Language Models Faithfully Express Their Intrinsic Uncertainty in Words?cited by 87
  25. 2024Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systemscited by 87
  26. 2023How FaR Are Large Language Models From Agents with Theory-of-Mind?cited by 76
  27. 2024LLMs achieve adult human performance on higher-order theory of mind taskscited by 61
  28. 2023The Challenge of Using LLMs to Simulate Human Behavior: A Causal Inference Perspectivecited by 59
  29. 2022Quantifying Social Biases Using Templates is Unreliablecited by 39
  30. 2023Llamas Know What GPTs Don't Show: Surrogate Models for Confidence Estimationcited by 39
  31. 2024Improving Uncertainty Estimation through Semantically Diverse Language Generationcited by 35
  32. 2022The Tail Wagging the Dog: Dataset Construction Biases of Social Bias Benchmarkscited by 31
  33. 2024Theory of Mind abilities of Large Language Models in Human-Robot Interaction : An Illusion?cited by 22
  34. 2024From Loops to Oops: Fallback Behaviors of Language Models Under Uncertaintycited by 14
  35. 2024An Evaluation of Estimative Uncertainty in Large Language Modelscited by 13
  36. 2024LLMs' Reading Comprehension Is Affected by Parametric Knowledge and Struggles with Hypothetical Statementscited by 12
  37. 2022Probing neural language models for understanding of words of estimative probabilitycited by 7
  38. 2024Using LLMs to Model the Beliefs and Preferences of Targeted Populationscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 8; 2026: 6.

15 papers.

Sort:

By area:

  1. 2025Bayesian Teaching Enables Probabilistic Reasoning in Large Language Modelscited by 37
  2. 2025Position: Uncertainty Quantification Needs Reassessment for Large-language Model Agentscited by 32
  3. 2025Metacognition and Uncertainty Communication in Humans and Large Language Modelscited by 23
  4. 2025Revisiting Uncertainty Estimation and Calibration of Large Language Modelscited by 23
  5. 2024Epistemic Integrity in Large Language Modelscited by 7
  6. 2025Can Large Language Models Express Uncertainty Like Human?cited by 5
  7. 2025Rethinking Prospect Theory for LLMs: Revealing the Instability of Decision-Making under Epistemic Uncertaintycited by 4
  8. 2026Can LLMs Use Linguistic Uncertainty Markers to Reliably Reflect Intrinsic Confidence?cited by 2
  9. 2025Evaluating Large Language Models for Evidence-Based Clinical Question Answeringcited by 1
  10. 2026Humans and LLMs Diverge on Probabilistic Inferencescited by 1
  11. 2025Human-Alignment and Calibration of Inference-Time Uncertainty in Large Language Modelscited by 0
  12. 2026Uncertainty Quantification for LLM Agents: A Taxonomy, an Evaluation Protocol, and an Empirical Studycited by 0
3 more
  1. 2026Calibration, Not Answer Selection: Distilling Internal Confidence in Reasoning Modelscited by 0
  2. 2026"very likely" Means "uncertain"? How LLMs Diverge from Humans in Linguistic Uncertainty Quantificationcited by 0
  3. 2026Bayesian Fine-tuning Yields Language Models that are as Bayesian as their Beliefs Allowcited by 0