mlsanityData to 7 Oct 2026

Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertainty

arXiv 2401.06730 ↗First posted 12 Jan 2024cs.CL cs.AI cs.HCcited by 106

Not on the map. Why?

Abstract (arXiv)

As natural language becomes the default interface for human-AI interaction, there is a need for LMs to appropriately communicate uncertainties in downstream applications. In this work, we investigate how LMs incorporate confidence in responses via natural language and how downstream users behave in response to LM-articulated uncertainties. We examine publicly deployed models and find that LMs are reluctant to express uncertainties when answering questions even when they produce incorrect responses. LMs can be explicitly prompted to express confidences, but tend to be overconfident, resulting in high error rates (an average of 47%) among confident responses. We test the risks of LM overconfidence by conducting human experiments and show that users rely heavily on LM generations, whether or not they are marked by certainty. Lastly, we investigate the preference-annotated datasets used in post training alignment and find that humans are biased against texts with uncertainty. Our work highlights new safety harms facing human-LM interactions and proposes design recommendations and mitigating strategies moving forward.

Key facts. Cited by 106 papers in the map (data through 7 Oct 2026). Builds on 37 of its 98 references that are in the map. First posted on arXiv 12 Jan 2024.

Builds on

37 of 98 references.

References · 37

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  8. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  9. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  10. 2017Explanation in Artificial Intelligence: Insights from the Social Sciencescited by 1,201
  11. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  12. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
25 more
  1. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  2. 2023Whose Opinions Do Language Models Reflect?cited by 759
  3. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  4. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
  5. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 439
  6. 2020Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performancecited by 396
  7. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  8. 2019Using Pre-Training Can Improve Model Robustness and Uncertaintycited by 373
  9. 2020Calibration of Pre-trained Transformerscited by 307
  10. 2021All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Textcited by 287
  11. 2021Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detectioncited by 258
  12. 2018Manipulating and Measuring Model Interpretabilitycited by 256
  13. 2020Selective Question Answering under Domain Shiftcited by 228
  14. 2022Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMscited by 220
  15. 2020Reducing conversational agents' overconfidence through linguistic calibrationcited by 209
  16. 2021Expanding Explainability: Towards Social Transparency in AI systemscited by 124
  17. 2019Human-Centered Tools for Coping with Imperfect Algorithms during Medical Decision-Makingcited by 109
  18. 2022Evaluating Human-Language Model Interactioncited by 90
  19. 2020A Case for Humans-in-the-Loop: Decisions in the Presence of Erroneous Algorithmic Scorescited by 74
  20. 2023Cognitive Dissonance: Why Do Language Model Outputs Disagree with Internal Representations of Truthfulness?cited by 62
  21. 2022A Close Look into the Calibration of Pre-trained Language Modelscited by 57
  22. 2020Conceptual Metaphors Impact Perceptions of Human-AI Collaborationcited by 47
  23. 2023AlignDiff: Aligning Diverse Human Preferences via Behavior-Customisable Diffusion Modelcited by 37
  24. 2023A Diachronic Perspective on User Trust in AI under Uncertaintycited by 23
  25. 2017Are distributional representations ready for the real world? Evaluating word vectors for grounded perceptual meaningcited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 35; 2025: 35; 2026: 36.

106 papers.

Sort:

By area:

  1. 2024Don't Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaborationcited by 176
  2. 2024A Roadmap to Pluralistic Alignmentcited by 175
  3. 2025Process Reward Models That Thinkcited by 118
  4. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 116
  5. 2024Rethinking Interpretability in the Era of Large Language Modelscited by 110
  6. 2024What Large Language Models Know and What People Think They Knowcited by 109
  7. 2024SaySelf: Teaching LLMs to Express Confidence with Self-Reflective Rationalescited by 102
  8. 2024Modeling Future Conversation Turns to Teach LLMs to Ask Clarifying Questionscited by 70
  9. 2024Calibrating Large Language Models Using Their Generations Onlycited by 63
  10. 2025Reasoning Models Better Express Their Confidencecited by 63
  11. 2024Uncertainty-Based Abstention in LLMs Improves Safety and Reduces Hallucinationscited by 51
  12. 2025Calibrating Verbal Uncertainty as a Linear Feature to Reduce Hallucinationscited by 48
38 more
  1. 2024Can Unconfident LLM Annotations Be Used for Confident Conclusions?cited by 44
  2. 2025Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistenciescited by 39
  3. 2025Mind the Confidence Gap: Overconfidence, Calibration, and Distractor Effects in Large Language Modelscited by 39
  4. 2024Securing Large Language Models: Threats, Vulnerabilities and Responsible Practicescited by 36
  5. 2024Knowledge Boundary of Large Language Models: A Surveycited by 36
  6. 2024Are LLM-Judges Robust to Expressions of Uncertainty? Investigating the effect of Epistemic Markers on LLM-based Evaluationcited by 29
  7. 2024Fact-and-Reflection (FaR) Improves Confidence Calibration of Large Language Modelscited by 27
  8. 2024Fundamental Problems With Model Editing: How Should Rational Belief Revision Work in LLMs?cited by 24
  9. 2024On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMscited by 24
  10. 2025Metacognition and Uncertainty Communication in Humans and Large Language Modelscited by 23
  11. 2025What's In My Human Feedback? Learning Interpretable Descriptions of Preference Datacited by 22
  12. 2024Towards Trustworthy AI: A Review of Ethical and Robust Large Language Modelscited by 21
  13. 2024A Survey on the Honesty of Large Language Modelscited by 21
  14. 2025Rejected Dialects: Biases Against African American Language in Reward Modelscited by 20
  15. 2024Probabilities of Chat LLMs Are Miscalibrated but Still Predict Correctness on Multiple-Choice Q&Acited by 16
  16. 2025Revisiting Epistemic Markers in Confidence Estimation: Can Markers Accurately Reflect Large Language Models' Uncertainty?cited by 16
  17. 2024Rel-A.I.: An Interaction-Centered Approach To Measuring Human-LM Reliancecited by 15
  18. 2024Perceptions of Linguistic Uncertainty by Language Models and Humanscited by 15
  19. 2025MetaFaith: Faithful Natural Language Uncertainty Expression in LLMscited by 15
  20. 2026Ask or Assume? Uncertainty-Aware Clarification-Seeking in Coding Agentscited by 15
  21. 2024Decoding the AI Pen: Techniques and Challenges in Detecting AI-Generated Textcited by 14
  22. 2024LACIE: Listener-Aware Finetuning for Confidence Calibration in Large Language Modelscited by 14
  23. 2024Teaching LLMs to Abstain across Languages via Multilingual Feedbackcited by 14
  24. 2025Don't lie to your friends: Learning what you know from collaborative self-playcited by 14
  25. 2025Anthropomimetic Uncertainty: What Verbalized Uncertainty in Language Models is Missingcited by 14
  26. 2025Enhancing LLM Reliability via Explicit Knowledge Boundary Modelingcited by 13
  27. 2025Uncertainty Distillation: Teaching Language Models to Express Semantic Confidencecited by 13
  28. 2024LLMs' Reading Comprehension Is Affected by Parametric Knowledge and Struggles with Hypothetical Statementscited by 12
  29. 2024Natural Language Processing RELIES on Linguisticscited by 9
  30. 2024Are language models rational? The case of coherence norms and belief revisioncited by 9
  31. 2025FinTrust: A Comprehensive Benchmark of Trustworthiness Evaluation in Finance Domaincited by 8
  32. 2024Epistemic Integrity in Large Language Modelscited by 7
  33. 2025Advancing the Scientific Method with Large Language Models: From Hypothesis to Discoverycited by 7
  34. 2024Search Engines Post-ChatGPT: How Generative Artificial Intelligence Could Make Search Less Reliablecited by 6
  35. 2024Rethinking the Uncertainty: A Critical Review and Analysis in the Era of Large Language Modelscited by 6
  36. 2024Reliable Decision Making via Calibration Oriented Retrieval Augmented Generationcited by 6
  37. 2025Adjust for Trust: Mitigating Trust-Induced Inappropriate Reliance on AI Assistancecited by 6
  38. 2024Beyond Scores: A Modular RAG-Based System for Automatic Short Answer Scoring with Feedbackcited by 5

50 most cited of 106 All 106 (JSON)