mlsanityData to 7 Oct 2026

Probabilities of Chat LLMs Are Miscalibrated but Still Predict Correctness on Multiple-Choice Q&A

arXiv 2402.13213 ↗First posted 20 Feb 2024cs.CL cs.AI cs.LGcited by 16

Not on the map. Why?

Abstract (arXiv)

We study 15 large language models (LLMs) fine-tuned for chat and find that their maximum softmax probabilities (MSPs) are consistently miscalibrated on multiple-choice Q&A. However, those MSPs might still encode useful uncertainty information. Specifically, we hypothesized that wrong answers would be associated with smaller MSPs compared to correct answers. Via rigorous statistical testing, we show that this hypothesis holds for models which perform well on the underlying Q&A task. We also find a strong direction correlation between Q&A accuracy and MSP correctness prediction, while finding no correlation between Q&A accuracy and calibration error. This suggests that within the current fine-tuning paradigm, we can expect correctness prediction but not calibration to improve as LLM capabilities progress. To demonstrate the utility of correctness prediction, we show that when models have the option to abstain, performance can be improved by selectively abstaining based on the MSP of the initial model response, using only a small amount of labeled data to choose the MSP threshold.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 32 of its 66 references that are in the map. First posted on arXiv 20 Feb 2024.

Builds on

32 of 66 references.

References · 32

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2017On Calibration of Modern Neural Networkscited by 4,860
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  10. 2023Mistral 7Bcited by 3,065
  11. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  12. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
20 more
  1. 2024Mixtral of Expertscited by 1,785
  2. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  3. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  4. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  5. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  6. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  7. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  8. 2019Scaling Out-of-Distribution Detection for Real-World Settingscited by 465
  9. 2020Calibration of Pre-trained Transformerscited by 307
  10. 2020Pretrained Transformers Improve Out-of-Distribution Robustnesscited by 293
  11. 2023Generating with Confidence: Uncertainty Quantification for Black-box Large Language Modelscited by 286
  12. 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 253
  13. 2023Do Large Language Models Know What They Don't Know?cited by 238
  14. 2020Reducing conversational agents' overconfidence through linguistic calibrationcited by 209
  15. 2023SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scalingcited by 172
  16. 2024Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertaintycited by 106
  17. 2015Posterior calibration and exploratory analysis for natural language processing modelscited by 75
  18. 2023On the Calibration of Large Language Models and Alignmentcited by 72
  19. 2023Uncertainty in Natural Language Processing: Sources, Quantification, and Applicationscited by 44
  20. 2023Investigating Uncertainty Calibration of Aligned Language Models under the Multiple-Choice Settingcited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 6; 2026: 6.

16 papers.

Sort:

By area:

  1. 2024Uncertainty Estimation and Quantification for LLMs: A Simple Supervised Approachcited by 73
  2. 2024Rethinking Uncertainty Estimation in LLMs: A Principled Single-Sequence Measurecited by 18
  3. 2024Uncertainty is Fragile: Manipulating Uncertainty in Large Language Modelscited by 13
  4. 2025Check Yourself Before You Wreck Yourself: Selectively Quitting Improves LLM Agent Safetycited by 12
  5. 2026Causal Evidence that Language Models use Confidence to Drive Behaviorcited by 9
  6. 2025TraPO: A Semi-Supervised Reinforcement Learning Framework for Boosting LLM Reasoningcited by 6
  7. 2025Calibrating LLM Judges: Linear Probes for Fast and Reliable Uncertainty Estimationcited by 6
  8. 2024Efficiently Deploying LLMs with Controlled Riskcited by 5
  9. 2025Robust Decision Making with Partially Calibrated Forecastscited by 3
  10. 2026PRISM: Probing Reasoning, Instruction, and Source Memory in LLM Hallucinationscited by 3
  11. 2025Prompt Stability in Code LLMs: Measuring Sensitivity across Emotion- and Personality-Driven Variationscited by 1
  12. 2025Efficient Uncertainty Estimation for LLM-based Entity Linking in Tabular Datacited by 0
4 more
  1. 2026Task-Aware Calibration: Provably Optimal Decoding in LLMscited by 0
  2. 2026LLM-Powered Virtual Population for Demand Simulation and Pricingcited by 0
  3. 2026There Is No Neutral Harness: Modern LLM Leaderboards Are Manufactured by Config-Fragile Itemscited by 0
  4. 2026Pinocchio: Fast Uncertainty Estimates for Black-Box Language Modelscited by 0