mlsanityData to 7 Oct 2026

Navigating the Grey Area: How Expressions of Uncertainty and Overconfidence Affect Language Models

arXiv 2302.13439 ↗First posted 26 Feb 2023cs.CL cs.AIcited by 91

Not on the map. Why?

Abstract (arXiv)

The increased deployment of LMs for real-world tasks involving knowledge and facts makes it important to understand model epistemology: what LMs think they know, and how their attitudes toward that knowledge are affected by language use in their inputs. Here, we study an aspect of model epistemology: how epistemic markers of certainty, uncertainty, or evidentiality like "I'm sure it's", "I think it's", or "Wikipedia says it's" affect models, and whether they contribute to model failures. We develop a typology of epistemic markers and inject 50 markers into prompts for question answering. We find that LMs are highly sensitive to epistemic markers in prompts, with accuracies varying more than 80%. Surprisingly, we find that expressions of high certainty result in a 7% decrease in accuracy as compared to low certainty expressions; similarly, factive verbs hurt performance, while evidentials benefit performance. Our analysis of a popular pretraining dataset shows that these markers of uncertainty are associated with answers on question-answering websites, while markers of certainty are associated with questions. These associations may suggest that the behavior of LMs is based on mimicking observed language use, rather than truly reflecting epistemic uncertainty.

Key facts. Cited by 91 papers in the map (data through 7 Oct 2026). Builds on 19 of its 51 references that are in the map. First posted on arXiv 26 Feb 2023.

Builds on

19 of 51 references.

References · 19

  1. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  2. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  3. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  4. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  5. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  6. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 439
  7. 2021Revisiting the Calibration of Modern Neural Networkscited by 387
  8. 2020Calibration of Pre-trained Transformerscited by 307
  9. 2022Social Simulacra: Creating Populated Prototypes for Social Computing Systemscited by 300
  10. 2020Selective Question Answering under Domain Shiftcited by 228
  11. 2022Demystifying Prompts in Language Models via Perplexity Estimationcited by 215
  12. 2019Automatically Neutralizing Subjective Bias in Textcited by 105
7 more
  1. 2020On the Inference Calibration of Neural Machine Translationcited by 61
  2. 2022Prompt-and-Rerank: A Method for Zero-Shot and Few-Shot Arbitrary Textual Style Transfer with Small Language Modelscited by 49
  3. 2022Uncertainty Estimation for Language Reward Modelscited by 35
  4. 2022Calibrated Interpretation: Confidence Estimation in Semantic Parsingcited by 33
  5. 2018Neural models of factualitycited by 20
  6. 2021The Possible, the Plausible, and the Desirable: Event-Based Modality Detection for Language Processingcited by 13
  7. 2021He Thinks He Knows Better than the Doctors: BERT for Event Factuality Fails on Pragmaticscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 26; 2025: 43; 2026: 20.

91 papers.

Sort:

By area:

  1. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  2. 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 253
  3. 2024Don't Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaborationcited by 176
  4. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 116
  5. 2024What Large Language Models Know and What People Think They Knowcited by 109
  6. 2024SaySelf: Teaching LLMs to Express Confidence with Self-Reflective Rationalescited by 102
  7. 2024On Verbalized Confidence Scores for LLMscited by 83
  8. 2024"I'm Not Sure, But...": Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trustcited by 67
  9. 2024Learning to Route LLMs with Confidence Tokenscited by 66
  10. 2025Missing Premise exacerbates Overthinking: Are Reasoning Models losing Critical Thinking Skill?cited by 58
  11. 2025A Survey of Uncertainty Estimation Methods on Large Language Modelscited by 57
  12. 2024Calibration and Correctness of Language Models for Codecited by 49
38 more
  1. 2025Calibrating Verbal Uncertainty as a Linear Feature to Reduce Hallucinationscited by 48
  2. 2024Improving Uncertainty Estimation through Semantically Diverse Language Generationcited by 34
  3. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  4. 2024Fact-and-Reflection (FaR) Improves Confidence Calibration of Large Language Modelscited by 27
  5. 2024Confidence Calibration and Rationalization for LLMs via Multi-Agent Deliberationcited by 26
  6. 2024Finetuning Language Models to Emit Linguistic Expressions of Uncertaintycited by 25
  7. 2024Deconstructing The Ethics of Large Language Models from Long-standing Issues to New-emerging Dilemmas: A Surveycited by 23
  8. 2024Scalable Qualitative Coding with LLMs: Chain-of-Thought Reasoning Matches Human Performance in Some Hermeneutic Taskscited by 22
  9. 2024A Survey on the Honesty of Large Language Modelscited by 21
  10. 2024Aligning Large Language Models by On-Policy Self-Judgmentcited by 20
  11. 2025An Empirical Analysis of Uncertainty in Large Language Model Evaluationscited by 20
  12. 2025SteerConf: Steering LLMs for Confidence Elicitationcited by 20
  13. 2025Robust Hallucination Detection in LLMs via Adaptive Token Selectioncited by 19
  14. 2024Rethinking Uncertainty Estimation in LLMs: A Principled Single-Sequence Measurecited by 18
  15. 2025Disentangling Reasoning and Knowledge in Medical Large Language Modelscited by 17
  16. 2025Overconfidence in LLM-as-a-Judge: Diagnosis and Confidence-Driven Solutioncited by 17
  17. 2025Revisiting Epistemic Markers in Confidence Estimation: Can Markers Accurately Reflect Large Language Models' Uncertainty?cited by 16
  18. 2025MetaFaith: Faithful Natural Language Uncertainty Expression in LLMscited by 15
  19. 2024LACIE: Listener-Aware Finetuning for Confidence Calibration in Large Language Modelscited by 14
  20. 2024Teaching LLMs to Abstain across Languages via Multilingual Feedbackcited by 14
  21. 2025Anthropomimetic Uncertainty: What Verbalized Uncertainty in Language Models is Missingcited by 14
  22. 2024UAlign: Leveraging Uncertainty Estimations for Factuality Alignment on Large Language Modelscited by 13
  23. 2025UNCLE: Benchmarking Uncertainty Expressions in Long-Form Generationcited by 11
  24. 2025MMBoundary: Advancing MLLM Knowledge Boundary Awareness through Reasoning Step Confidence Calibrationcited by 11
  25. 2025When Two LLMs Debate, Both Think They'll Wincited by 10
  26. 2025Towards Adaptive Memory-Based Optimization for Enhanced Retrieval-Augmented Generationcited by 9
  27. 2026Accommodation and Epistemic Vigilance: A Pragmatic Account of Why LLMs Fail to Challenge Harmful Beliefscited by 8
  28. 2024Epistemic Integrity in Large Language Modelscited by 7
  29. 2024Text-to-SQL Calibration: No Need to Ask -- Just Rescale Model Probabilitiescited by 7
  30. 2026The Illusion of Certainty: Decoupling Capability and Calibration in On-Policy Distillationcited by 6
  31. 2025GRACE: A Granular Benchmark for Evaluating Model Calibration against Human Calibrationcited by 5
  32. 2025From Evidence to Belief: A Bayesian Epistemology Approach to Language Modelscited by 5
  33. 2025Mind the Generation Process: Fine-Grained Confidence Estimation During LLM Generationcited by 5
  34. 2025Don't Think Twice! Over-Reasoning Impairs Confidence Calibrationcited by 5
  35. 2025Robust Sampling for Active Statistical Inferencecited by 5
  36. 2025On LLMs' Internal Representation of Code Correctnesscited by 5
  37. 2024MlingConf: A Comprehensive Study of Multilingual Confidence Estimation on Large Language Modelscited by 4
  38. 2025Learning from Reference Answers: Versatile Language Model Alignment without Binary Human Preference Datacited by 4

50 most cited of 91 All 91 (JSON)