# Verbalizing LLM's Higher-order Uncertainty via Imprecise Probabilities

arXiv: https://arxiv.org/abs/2603.10396 · HTML: https://mlsanity.com/paper/2603.10396/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 66 references that are in the map.
- First posted on arXiv 11 Mar 2026.

## Builds on (most cited first, 30 of 49)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs](https://mlsanity.com/paper/2306.13063/) (2023)
- [An Explanation of In-context Learning as Implicit Bayesian Inference](https://mlsanity.com/paper/2111.02080/) (2021)
- [Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback](https://mlsanity.com/paper/2305.14975/) (2023)
- [Teaching Models to Express Their Uncertainty in Words](https://mlsanity.com/paper/2205.14334/) (2022)
- [Transformers learn in-context by gradient descent](https://mlsanity.com/paper/2212.07677/) (2022)
- [AmbigQA: Answering Ambiguous Open-domain Questions](https://mlsanity.com/paper/2004.10645/) (2020)
- [tinyBenchmarks: evaluating LLMs with fewer examples](https://mlsanity.com/paper/2402.14992/) (2024)
- [Prompting GPT-3 To Be Reliable](https://mlsanity.com/paper/2210.09150/) (2022)
- [ShinkaEvolve: Towards Open-Ended And Sample-Efficient Program Evolution](https://mlsanity.com/paper/2509.19349/) (2025)
- [A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directions](https://mlsanity.com/paper/2412.05563/) (2024)
- [Decomposing Uncertainty for Large Language Models through Input Clarification Ensembling](https://mlsanity.com/paper/2311.08718/) (2023)
- [Reinforcement Learning with Rubric Anchors](https://mlsanity.com/paper/2508.12790/) (2025)
- [SaySelf: Teaching LLMs to Express Confidence with Self-Reflective Rationales](https://mlsanity.com/paper/2405.20974/) (2024)
- [Are self-explanations from Large Language Models faithful?](https://mlsanity.com/paper/2401.07927/) (2024)
- [A Survey of Uncertainty Estimation Methods on Large Language Models](https://mlsanity.com/paper/2503.00172/) (2025)
- [Is In-Context Learning in Large Language Models Bayesian? A Martingale Perspective](https://mlsanity.com/paper/2406.00793/) (2024)
- [SPUQ: Perturbation-Based Uncertainty Quantification for Large Language Models](https://mlsanity.com/paper/2403.02509/) (2024)
- [Uncertainty-Based Abstention in LLMs Improves Safety and Reduces Hallucinations](https://mlsanity.com/paper/2404.10960/) (2024)
- [Large Language Models Assume People are More Rational than We Really are](https://mlsanity.com/paper/2406.17055/) (2024)
- [Distinguishing the Knowable from the Unknowable with Language Models](https://mlsanity.com/paper/2402.03563/) (2024)
- [Walk the Talk? Measuring the Faithfulness of Large Language Model Explanations](https://mlsanity.com/paper/2504.14150/) (2025)
- [Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens](https://mlsanity.com/paper/2508.01191/) (2025)
- [ConfTuner: Training Large Language Models to Express Their Confidence Verbally](https://mlsanity.com/paper/2508.18847/) (2025)

## Built on by (most cited first, 5 of 5)

- [Open-Ended Task Discovery via Bayesian Optimization](https://mlsanity.com/paper/2605.07572/) (2026)
- [Credal Large Language Models for Semantic Commitment under Uncertainty](https://mlsanity.com/paper/2608.23244/) (2026)
- [Regret Analysis of Guided Diffusion for Black-Box Optimization over Structured Inputs](https://mlsanity.com/paper/2605.10385/) (2026)
- [From Answers to Interpretations: Rethinking Ambiguity-Induced Aleatoric Uncertainty Estimation in LLMs](https://mlsanity.com/paper/2609.04543/) (2026)
- [XU-RS: Explaining Credal Width in Random-Set Language Models](https://mlsanity.com/paper/2609.37594/) (2026)

## Abstract (arXiv)

Despite the growing demand for eliciting uncertainty from large language models (LLMs), empirical evidence suggests that LLM behavior is not always adequately captured by the elicitation techniques developed under the classical probabilistic uncertainty framework. This mismatch leads to systematic failure modes, particularly in settings that involve ambiguous question-answering, in-context learning, and self-reflection. To address this, we propose novel prompt-based uncertainty elicitation techniques grounded in imprecise probabilities, a principled framework for repesenting and eliciting higher-order uncertainty. Here, first-order uncertainty captures uncertainty over possible responses to a prompt, while second-order uncertainty (uncertainty about uncertainty) quantifies indeterminacy in the underlying probability model itself. We introduce general-purpose prompting and post-processing procedures to directly elicit and quantify both orders of uncertainty, and demonstrate their effectiveness across diverse settings. Our approach enables more faithful uncertainty reporting from LLMs, improving credibility and supporting downstream decision-making.