# From Calibration to Collaboration: LLM Uncertainty Quantification Should Be More Human-Centered

arXiv: https://arxiv.org/abs/2506.07461 · HTML: https://mlsanity.com/paper/2506.07461/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 101 of its 130 references that are in the map.
- First posted on arXiv 9 Jun 2025.

## Builds on (most cited first, 30 of 101)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs](https://mlsanity.com/paper/2306.13063/) (2023)
- [Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods](https://mlsanity.com/paper/1910.09457/) (2019)
- [Selective Classification for Deep Neural Networks](https://mlsanity.com/paper/1705.08500/) (2017)
- [Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation](https://mlsanity.com/paper/2302.09664/) (2023)
- [Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback](https://mlsanity.com/paper/2305.14975/) (2023)
- [SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models](https://mlsanity.com/paper/2303.08896/) (2023)
- [Teaching Models to Express Their Uncertainty in Words](https://mlsanity.com/paper/2205.14334/) (2022)
- [Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies](https://mlsanity.com/paper/2101.02235/) (2021)
- [Latent Retrieval for Weakly Supervised Open Domain Question Answering](https://mlsanity.com/paper/1906.00300/) (2019)
- [WildChat: 1M ChatGPT Interaction Logs in the Wild](https://mlsanity.com/paper/2405.01470/) (2024)
- [Large Language Models Are Not Robust Multiple Choice Selectors](https://mlsanity.com/paper/2309.03882/) (2023)
- [ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection](https://mlsanity.com/paper/2203.09509/) (2022)
- [AmbigQA: Answering Ambiguous Open-domain Questions](https://mlsanity.com/paper/2004.10645/) (2020)
- [Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models](https://mlsanity.com/paper/2305.19187/) (2023)
- [Bridging Language and Items for Retrieval and Recommendation: Benchmarking LLMs as Semantic Encoders](https://mlsanity.com/paper/2403.03952/) (2024)
- [Challenges and Strategies in Cross-Cultural NLP](https://mlsanity.com/paper/2203.10020/) (2022)
- [WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild](https://mlsanity.com/paper/2406.04770/) (2024)
- [Reducing conversational agents' overconfidence through linguistic calibration](https://mlsanity.com/paper/2012.14983/) (2020)
- [Human-Centered Artificial Intelligence: Reliable, Safe & Trustworthy](https://mlsanity.com/paper/2002.04087/) (2020)
- [Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similarities](https://mlsanity.com/paper/2405.20003/) (2024)
- [What Can We Learn from Collective Human Opinions on Natural Language Inference Data?](https://mlsanity.com/paper/2010.03532/) (2020)
- [Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Models](https://mlsanity.com/paper/2307.01379/) (2023)
- [SituatedQA: Incorporating Extra-Linguistic Contexts into QA](https://mlsanity.com/paper/2109.06157/) (2021)
- [Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systems](https://mlsanity.com/paper/2001.08298/) (2020)

## Built on by (most cited first, 21 of 21)

- [Trace Length is a Simple Uncertainty Signal in Reasoning Models](https://mlsanity.com/paper/2510.10409/) (2025)
- [The Illusion of Certainty: Uncertainty Quantification for LLMs Fails under Ambiguity](https://mlsanity.com/paper/2511.04418/) (2025)
- [Are LLM Decisions Faithful to Verbal Confidence?](https://mlsanity.com/paper/2601.07767/) (2026)
- [Can Large Language Models Express Uncertainty Like Human?](https://mlsanity.com/paper/2509.24202/) (2025)
- [On the Robustness of Verbal Confidence of LLMs in Adversarial Attacks](https://mlsanity.com/paper/2507.06489/) (2025)
- [Localized Calibrated Uncertainty in Code Language Models](https://mlsanity.com/paper/2512.24560/) (2025)
- [From Passive Metric to Active Signal: The Evolving Role of Uncertainty Quantification in Large Language Models](https://mlsanity.com/paper/2601.15690/) (2026)
- [SAFECAST: Robust Failure Detection for VLA Policies with Contrast-Set Training and Calibration](https://mlsanity.com/paper/2608.04246/) (2026)
- [Uncertainty and Fairness Awareness in LLM-Based Recommendation Systems](https://mlsanity.com/paper/2602.02582/) (2026)
- [Learning to Decide with AI Assistance under Human-Alignment](https://mlsanity.com/paper/2605.12646/) (2026)
- [Testing LLM Arithmetic Reasoning Generalization with Automatic Numeric-Remapping Attacks](https://mlsanity.com/paper/2606.03606/) (2026)
- [LLM-based Agents for Forecasting and Prediction: Methods, Training, Evaluation, and Applications](https://mlsanity.com/paper/2608.23058/) (2026)
- [PaTAS: A Framework for Trust Propagation in Neural Networks Using Subjective Logic](https://mlsanity.com/paper/2511.20586/) (2025)
- [Knowing Your Uncertainty -- On the application of LLM in social sciences](https://mlsanity.com/paper/2512.05461/) (2025)
- [From Performance to Purpose: A Sociotechnical Taxonomy for Evaluating Large Language Model Utility](https://mlsanity.com/paper/2602.20513/) (2026)
- [Position: Uncertainty Quantification in LLMs is Just Unsupervised Clustering](https://mlsanity.com/paper/2605.19220/) (2026)
- [How Do Uncertainty Estimators Align with Hallucination Occurrence in LLMs?](https://mlsanity.com/paper/2605.27016/) (2026)
- [Localizing Input Uncertainty Quantification for Large Language Models via Shapley Values](https://mlsanity.com/paper/2605.28170/) (2026)
- [Clustered Self-Assessment: A Simple yet Effective Method for Uncertainty Quantification in Large Language Models](https://mlsanity.com/paper/2606.03846/) (2026)
- [Probability is Not Enough: Exploring and Counting Divergent Tokens for Reasoning Uncertainty Quantification in LLMs](https://mlsanity.com/paper/2609.38070/) (2026)
- [Characterizing Overconfident Failure in LLM-Based Code Generation](https://mlsanity.com/paper/2610.11300/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are increasingly assisting users in the real world, yet their reliability remains a concern. Uncertainty quantification (UQ) has been heralded as a tool to enhance human-LLM collaboration by enabling users to know when to trust LLM predictions. We argue that current practices for uncertainty quantification in LLMs are not optimal for developing useful UQ for human users making decisions in real-world tasks. Through an analysis of 40 LLM UQ methods, we identify three prevalent practices hindering the community's progress toward its goal of benefiting downstream users: 1) evaluating on benchmarks with low ecological validity; 2) considering only epistemic uncertainty; and 3) optimizing metrics that are not necessarily indicative of downstream utility. For each issue, we propose concrete user-centric practices and research directions that LLM UQ researchers should consider. Instead of hill-climbing on unrepresentative tasks using imperfect metrics, we argue that the community should adopt a more human-centered approach to LLM uncertainty quantification.