mlsanityData to 14 Oct 2026

QA-Calibration of Language Model Confidence Scores

arXiv 2410.06615 ↗First posted 9 Oct 2024cs.CL cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

To use generative question-and-answering (QA) systems for decision-making and in any critical application, these systems need to provide well-calibrated confidence scores that reflect the correctness of their answers. Existing calibration methods aim to ensure that the confidence score is, *on average*, indicative of the likelihood that the answer is correct. We argue, however, that this standard (average-case) notion of calibration is difficult to interpret for decision-making in generative QA. To address this, we generalize the standard notion of average calibration and introduce QA-calibration, which ensures calibration holds across different question-and-answer groups. We then propose discretized posthoc calibration schemes for achieving QA-calibration. We establish distribution-free guarantees on the performance of this method and validate our method on confidence scores returned by elicitation prompts across multiple QA benchmarks and large language models (LLMs).

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 32 of its 42 references that are in the map. First posted on arXiv 9 Oct 2024.

Builds on

32 of 42 references.

References · 32

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  5. 2017On Calibration of Modern Neural Networkscited by 4,960
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  7. 2023Mistral 7Bcited by 3,164
  8. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  9. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  10. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  11. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  12. 2022Language Models (Mostly) Know What They Knowcited by 1,856
20 more
  1. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  2. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  3. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
  4. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 819
  5. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
  6. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
  7. 2017Crowdsourcing Multiple Choice Science Questionscited by 765
  8. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
  9. 2016Inherent Trade-Offs in the Fair Determination of Risk Scorescited by 586
  10. 2017On Fairness and Calibrationcited by 406
  11. 2019Verified Uncertainty Calibrationcited by 309
  12. 2020Selective Question Answering under Domain Shiftcited by 238
  13. 2020Reducing conversational agents' overconfidence through linguistic calibrationcited by 210
  14. 2020Distribution-free binary classification: prediction sets, confidence intervals and calibrationcited by 79
  15. 2024Calibrating Large Language Models with Sample Consistencycited by 73
  16. 2024Calibrating Large Language Models Using Their Generations Onlycited by 65
  17. 2024Multicalibration for Confidence Scoring in LLMscited by 46
  18. 2021Distribution-free calibration guarantees for histogram binning without sample splittingcited by 45
  19. 2021Local Calibration: Metrics and Recalibrationcited by 24
  20. 2024Few-Shot Recalibration of Language Modelscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 9.

13 papers.

Sort:

By area:

  1. 2025From Calibration to Collaboration: LLM Uncertainty Quantification Should Be More Human-Centeredcited by 21
  2. 2025Trustworthy Medical Question Answering: An Evaluation-Centric Surveycited by 7
  3. 2026Calibrate, Don't Curate: Label-Efficient Estimation from Noisy LLM Judgescited by 5
  4. 2026Know More, Know Clearer: A Meta-Cognitive Framework for Knowledge Augmentation in Large Language Modelscited by 3
  5. 2026Calibrating Model-Based Evaluation Metrics for Summarizationcited by 3
  6. 2025HASHIRU: Hierarchical Agent System for Hybrid Intelligent Resource Utilizationcited by 2
  7. 2026LLM-Confidence Reranker: A Training-Free Approach for Enhancing Retrieval-Augmented Generation Systemscited by 2
  8. 2026Towards Trustworthy Report Generation: A Deep Research Agent with Progressive Confidence Estimation and Calibrationcited by 1
  9. 2025LACY: A Vision-Language Model-based Language-Action Cycle for Self-Improving Robotic Manipulationcited by 0
  10. 2026Assessing and Mitigating Miscalibration in LLM-Based Social Science Measurementcited by 0
  11. 2026Inducing Artificial Uncertainty in Language Modelscited by 0
  12. 2026RadFusion: Towards Threshold-Controllable Radiology Report Generationcited by 0
1 more
  1. 2026CHAIN: Calibrated LLM Forecasting via Causal-Temporal Hypergraph Inferencecited by 0