mlsanityData to 7 Oct 2026

On Calibration of Large Language Models: From Response To Capability

arXiv 2602.13540 ↗First posted 14 Feb 2026cs.CL cs.AI cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

Accurate confidence estimation is critical for reliable use of large language models (LLMs). Prior work on LLM calibration largely focuses on response-level confidence, which estimates the correctness of a single generated output. However, this formulation is misaligned with many practical settings where the central question is how likely a model is to solve a query overall. We show that this mismatch results from the stochastic nature of modern LLM decoding, under which single-response correctness fails to reflect underlying model capability. To address this issue, we introduce capability calibration, a new evaluation framework for measuring how well query-level confidence aligns with a model's expected accuracy on individual queries. We formally distinguish capability calibration (CC) from response calibration (RC) and show that the two differ both theoretically and empirically. We further show that CC is better suited than RC to applications like pass@k prediction and inference budget allocation. Finally, we evaluate common confidence estimation methods to understand the practical feasibility of CC.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 59 of its 74 references that are in the map. First posted on arXiv 14 Feb 2026.

Builds on

59 of 74 references.

References · 59

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2017On Calibration of Modern Neural Networkscited by 4,860
  9. 2023Let's Verify Step by Stepcited by 3,598
  10. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  11. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  12. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
38 more
  1. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  2. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  3. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  4. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  5. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  6. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  7. 2017Selective Classification for Deep Neural Networkscited by 799
  8. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  9. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
  10. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  11. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 787
  12. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  13. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 633
  14. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  15. 2024Measuring short-form factuality in large language modelscited by 345
  16. 2024Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Modelscited by 282
  17. 2024TrustLLM: Trustworthiness in Large Language Modelscited by 280
  18. 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 253
  19. 2020Selective Question Answering under Domain Shiftcited by 228
  20. 2021Implicit Representations of Meaning in Neural Language Modelscited by 179
  21. 2024The Effect of Sampling Temperature on Problem Solving in Large Language Modelscited by 168
  22. 2025Reasoning Models Know When They're Right: Probing Hidden States for Self-Verificationcited by 154
  23. 2023Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Modelscited by 147
  24. 2024A Thorough Examination of Decoding Methods in the Era of LLMscited by 112
  25. 2025Beyond Binary Rewards: Training LMs to Reason About Their Uncertaintycited by 107
  26. 2024Learning How Hard to Think: Input-Adaptive Allocation of LM Computationcited by 67
  27. 2024A Survey of Uncertainty Estimation in LLMs: Theory Meets Practicecited by 54
  28. 2023Adaptation with Self-Evaluation to Improve Selective Prediction in LLMscited by 49
  29. 2025SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledgecited by 44
  30. 2025How Do Large Language Monkeys Get Their Power (Laws)?cited by 40
  31. 2025Exploring the Impact of Temperature on Large Language Models:Hot or Cold?cited by 40
  32. 2025ConfTuner: Training Large Language Models to Express Their Confidence Verballycited by 37
  33. 2025No Answer Needed: Predicting LLM Answer Accuracy from Question-Only Linear Probescited by 35
  34. 2024Smoothie: Label Free Language Model Routingcited by 31
  35. 2025A Comprehensive Survey on the Trustworthiness of Large Language Models in Healthcarecited by 27
  36. 2024I Don't Know: Explicit Modeling of Uncertainty with an [IDK] Tokencited by 26
  37. 2025Beyond Random Sampling: Efficient Language Model Pretraining via Curriculum Learningcited by 25
  38. 2022Uncertainty Estimates of Predictions via a General Bias-Variance Decompositioncited by 20

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026An LLM-Native Psychometric Instrument Reveals a Self-Report--Behavior Gap Across 25 Modelscited by 4
  2. 2026Calibrated Confidence Estimation for Tabular Question Answeringcited by 0
  3. 2026Eta Given Delta: Defining LLM Tool Efficiency With Marginal Tool Utilitycited by 0
  4. 2026ProbGuard: Calibrated Safety Risk Estimation from LLM Output Distributionscited by 0
  5. 2026On the Pitfalls of Verbalized Confidence Priors for Calibrating Large Reasoning Modelscited by 0