mlsanityData to 14 Oct 2026

Improving the Calibration of Confidence Scores in Text Generation Using the Output Distribution's Characteristics

arXiv 2506.00637 ↗First posted 31 May 2025cs.CL cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

Well-calibrated model confidence scores can improve the usefulness of text generation models. For example, users can be prompted to review predictions with low confidence scores, to prevent models from returning bad or potentially dangerous predictions. However, confidence metrics are not always well calibrated in text generation. One reason is that in generation, there can be many valid answers, which previous methods do not always account for. Hence, a confident model could distribute its output probability among multiple sequences because they are all valid. We propose task-agnostic confidence metrics suited to generation, which rely solely on the probabilities associated with the model outputs without the need for further fine-tuning or heuristics. Using these, we are able to improve the calibration of BART and Flan-T5 on summarization, translation, and QA datasets.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 21 of its 35 references that are in the map. First posted on arXiv 31 May 2025.

Builds on

21 of 35 references.

References · 21

  1. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  2. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  3. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  4. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  5. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  6. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,292
  7. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
  8. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
  9. 2020Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awarenesscited by 389
  10. 2020Selective Question Answering under Domain Shiftcited by 238
  11. 2024Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similaritiescited by 159
  12. 2019Ensemble Distribution Distillationcited by 158
9 more
  1. 2018Correcting Length Bias in Neural Machine Translationcited by 120
  2. 2018Abstractive Summarization of Reddit Posts with Multi-level Memory Networkscited by 102
  3. 2019Calibration of Encoder Decoder Models for Neural Machine Translationcited by 68
  4. 2020DebateSum: A large-scale argument mining and summarization datasetcited by 34
  5. 2020Wat zei je? Detecting Out-of-Distribution Translations with Variational Transformerscited by 31
  6. 2023On Uncertainty Calibration and Selective Generation in Probabilistic Neural Summarization: A Benchmark Studycited by 25
  7. 2024Enhancing Confidence Expression in Large Language Models Through Learning from Past Experiencecited by 21
  8. 2022Combining Data Generation and Active Learning for Low-Resource Question Answeringcited by 5
  9. 2023Logit-Based Ensemble Distribution Distillation for Robust Autoregressive Sequence Uncertaintiescited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Sigmoid Head for Quality Estimation under Language Ambiguitycited by 0
  2. 2026Confident in a Confidence Score: Investigating the Sensitivity of Confidence Scores to Supervised Fine-Tuningcited by 0
  3. 2026Testing the Assumptions of Active Learning for Translation Tasks with Few Samplescited by 0
  4. 2026TRACE: Single-Pass Decoding-Trace Risk Localization for Generation Calibrationcited by 0