mlsanityData to 14 Oct 2026

Revisiting Epistemic Markers in Confidence Estimation: Can Markers Accurately Reflect Large Language Models' Uncertainty?

arXiv 2505.24778 ↗First posted 30 May 2025cs.CLcited by 16

Not on the map. Why?

Abstract (arXiv)

As large language models (LLMs) are increasingly used in high-stakes domains, accurately assessing their confidence is crucial. Humans typically express confidence through epistemic markers (e.g., "fairly confident") instead of numerical values. However, it remains unclear whether LLMs consistently use these markers to reflect their intrinsic confidence due to the difficulty of quantifying uncertainty associated with various markers. To address this gap, we first define marker confidence as the observed accuracy when a model employs an epistemic marker. We evaluate its stability across multiple question-answering datasets in both in-distribution and out-of-distribution settings for open-source and proprietary LLMs. Our results show that while markers generalize well within the same distribution, their confidence is inconsistent in out-of-distribution scenarios. These findings raise significant concerns about the reliability of epistemic markers for confidence estimation, underscoring the need for improved alignment between marker based confidence and actual model uncertainty. Our code is available at https://github.com/HKUST-KnowComp/MarConf.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 38 of its 69 references that are in the map. First posted on arXiv 30 May 2025.

Builds on

38 of 69 references.

References · 38

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2024Qwen2.5 Technical Reportcited by 4,942
  6. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  7. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  8. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  9. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  10. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 900
  11. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 767
  12. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
26 more
  1. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 743
  2. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 368
  3. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 323
  4. 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 256
  5. 2020Unsupervised Quality Estimation for Neural Machine Translationcited by 211
  6. 2021When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Datasetcited by 201
  7. 2023Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Modelscited by 150
  8. 2023Quantifying Uncertainty in Answers from any Language Model and Enhancing their Trustworthinesscited by 122
  9. 2024Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertaintycited by 106
  10. 2024Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraphcited by 105
  11. 2023Navigating the Grey Area: How Expressions of Uncertainty and Overconfidence Affect Language Modelscited by 91
  12. 2024Can Large Language Models Faithfully Express Their Intrinsic Uncertainty in Words?cited by 87
  13. 2024Self-Alignment for Factuality: Mitigating Hallucinations in LLMs via Self-Evaluationcited by 80
  14. 2023Uncertainty in Natural Language Processing: Sources, Quantification, and Applicationscited by 44
  15. 2024Think Twice Before Trusting: Self-Detection for Large Language Models through Comprehensive Answer Reflectioncited by 41
  16. 2024InternalInspector I2: Robust Confidence Estimation in LLMs through Internal Statescited by 30
  17. 2024Are LLM-Judges Robust to Expressions of Uncertainty? Investigating the effect of Epistemic Markers on LLM-based Evaluationcited by 29
  18. 2023CAR: Conceptualization-Augmented Reasoner for Zero-Shot Commonsense Question Answeringcited by 28
  19. 2024CANDLE: Iterative Conceptualization and Instantiation Distillation from Large Language Models for Commonsense Reasoningcited by 28
  20. 2023CAT: A Contextualized Conceptualization and Instantiation Framework for Commonsense Reasoningcited by 25
  21. 2024Comparing the Pearson and Spearman Correlation Coefficients Across Distributions and Sample Sizes: A Tutorial Using Simulations and Empirical Datacited by 16
  22. 2024Large Language Model Confidence Estimation via Black-Box Accesscited by 14
  23. 2024An Evaluation of Estimative Uncertainty in Large Language Modelscited by 13
  24. 2024MARS: Benchmarking the Metaphysical Reasoning Abilities of Language Models with a Multi-task Evaluation Datasetcited by 11
  25. 2024ComparisonQA: Evaluating Factuality Robustness of LLMs Through Knowledge Frequency Control and Uncertaintycited by 10
  26. 2025EcomScriptBench: A Multi-task Benchmark for E-commerce Script Planning via Step-wise Intention-Driven Product Associationcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 9.

16 papers.

Sort:

By area:

  1. 2025CostBench: Evaluating Multi-Turn Cost-Optimal Planning and Adaptation in Dynamic Environments for LLM Tool-Use Agentscited by 33
  2. 2025Trace Length is a Simple Uncertainty Signal in Reasoning Modelscited by 19
  3. 2025Anthropomimetic Uncertainty: What Verbalized Uncertainty in Language Models is Missingcited by 14
  4. 2026Why LLMs Aren't Scientists Yet: Lessons from Four Autonomous Research Attemptscited by 13
  5. 2025Diversity-Enhanced Reasoning for Subjective Questionscited by 9
  6. 2026NOVA: NOise-aware Verbal Confidence CAlibration for Robust Large Language Models in RAG Systemscited by 6
  7. 2025Rethinking Prospect Theory for LLMs: Revealing the Instability of Decision-Making under Epistemic Uncertaintycited by 4
  8. 2025Stop Before You Fail: Operational Capability Boundaries for Mitigating Unproductive Reasoning in Large Reasoning Modelscited by 4
  9. 2025CritiCal: Can Critique Help LLM Uncertainty or Confidence Calibration?cited by 4
  10. 2026Code2Math: Can Your Code Agent Effectively Evolve Math Problems Through Exploration?cited by 2
  11. 2026Can LLMs Use Linguistic Uncertainty Markers to Reliably Reflect Intrinsic Confidence?cited by 2
  12. 2026A Systematic Evaluation of Black-Box Uncertainty Estimation Methods for Large Language Modelscited by 2
4 more
  1. 2026Safer Policy Compliance with Dynamic Epistemic Fallbackcited by 1
  2. 2026How Uncertainty Estimation Scales with Sampling in Reasoning Modelscited by 1
  3. 2026Psi-Bench: Evaluating Persona-Sensitive Influencing in Persuasive Dialoguescited by 0
  4. 2026A Taxonomy of Cognitive Capability Gaps in Generative and Agentic AIcited by 0