mlsanityData to 14 Oct 2026

Improved Confidence Estimates for Black-Box Large Language Models

arXiv 2608.19323 ↗First posted 19 Aug 2026cs.LG cs.AI stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

Uncertainty quantification (UQ) is essential for the safe deployment of large language models (LLMs). Existing methods, from verbalized confidence to ones requiring multiple generations, are often zero-shot and produce scores quantifying uncertainty without the need for labelled data. Nonetheless, in practice one must always evaluate their performance on a dataset of interest before deployment. In this work we show that, by leveraging this dataset, we consistently outperform these existing scores. Specifically, we build simple classifiers that predict LLM response correctness by using these scores and the correctness of similar queries as features. Our method produces minimal computational overhead, making it a cheap and straightforward enhancement for UQ in LLMs for real-world applications.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 33 of its 50 references that are in the map. First posted on arXiv 19 Aug 2026.

Builds on

33 of 50 references.

References · 33

  1. 2017On Calibration of Modern Neural Networkscited by 4,960
  2. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  3. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  4. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  5. 2019Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methodscited by 913
  6. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 906
  7. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 900
  8. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 819
  9. 2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 792
  10. 2017Crowdsourcing Multiple Choice Science Questionscited by 765
  11. 2024Measuring short-form factuality in large language modelscited by 373
  12. 2024Hallucination is Inevitable: An Innate Limitation of Large Language Modelscited by 332
21 more
  1. 2023Generating with Confidence: Uncertainty Quantification for Black-box Large Language Modelscited by 293
  2. 2020Reducing conversational agents' overconfidence through linguistic calibrationcited by 210
  3. 2023Conformal Language Modelingcited by 170
  4. 2024Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similaritiescited by 159
  5. 2023Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Modelscited by 150
  6. 2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 121
  7. 2023Bayesian Low-rank Adaptation for Large Language Modelscited by 115
  8. 2023Decomposing Uncertainty for Large Language Models through Input Clarification Ensemblingcited by 114
  9. 2024Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertaintycited by 106
  10. 2024Theoretical Foundations of Conformal Predictioncited by 101
  11. 2023A Confederacy of Models: a Comprehensive Evaluation of LLMs on Creative Writingcited by 99
  12. 2024On Verbalized Confidence Scores for LLMscited by 83
  13. 2023LoRA ensembles for large language model fine-tuningcited by 67
  14. 2024Calibrating Large Language Models Using Their Generations Onlycited by 65
  15. 2024SPUQ: Perturbation-Based Uncertainty Quantification for Large Language Modelscited by 56
  16. 2024Rethinking Aleatoric and Epistemic Uncertaintycited by 42
  17. 2025Position: Uncertainty Quantification Needs Reassessment for Large-language Model Agentscited by 32
  18. 2024Improving Uncertainty Quantification in Large Language Models via Semantic Embeddingscited by 30
  19. 2023Data-Efficient Multimodal Fusion on a Single GPUcited by 11
  20. 2024Conformal Generative Modeling with Improved Sample Efficiency through Sequential Greedy Filteringcited by 10
  21. 2026Conf-Gen: Conformal Uncertainty Quantification for Generative Modelscited by 1

Built on by

Not cited yet.