mlsanityData to 14 Oct 2026

Confident in a Confidence Score: Investigating the Sensitivity of Confidence Scores to Supervised Fine-Tuning

arXiv 2604.08974 ↗First posted 10 Apr 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Uncertainty quantification techniques measure confidence in language model outputs to support critical applications like hallucination detection and selective prediction. While prior work has developed various confidence metrics and demonstrated their calibration for classification tasks or using verbalized confidence, the robustness of probability-based and self-consistency-based UQ metrics for natural language generation remains underexplored particularly under model adaptation. Since practitioners routinely apply supervised fine-tuning to adapt models to new tasks, a key question arises: do confidence metrics maintain their calibration when models are fine-tuned? We investigate this question across NLG tasks including translation, question answering, and mathematical reasoning. We find that calibration shifts substantially after SFT: across 216 configurations, it degrades in 112 cases and improves in 104, with confidence scores shifting due to factors beyond output quality, such as proximity to the training distribution. Degradation is therefore neither universal nor rare, and its direction cannot be anticipated from the pre-SFT model. Through a downstream task evaluation, we show that this miscalibration substantially reduces the practical utility of confidence scores for identifying correct answers. Our findings reveal that existing confidence metrics for NLG cannot be reliably deployed off-the-shelf after fine-tuning, highlighting the need for calibration-robust UQ methods under model adaptation.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 25 of its 40 references that are in the map. First posted on arXiv 10 Apr 2026.

Builds on

25 of 40 references.

References · 25

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  4. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  5. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  6. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  7. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
  8. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
  9. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
  10. 2020Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awarenesscited by 389
  11. 2020Selective Question Answering under Domain Shiftcited by 238
  12. 2024Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similaritiescited by 159
13 more
  1. 2019Ensemble Distribution Distillationcited by 158
  2. 2018Correcting Length Bias in Neural Machine Translationcited by 120
  3. 2024Taming Overconfidence in LLMs: Reward Calibration in RLHFcited by 112
  4. 2019Calibration of Encoder Decoder Models for Neural Machine Translationcited by 68
  5. 2020Wat zei je? Detecting Out-of-Distribution Translations with Variational Transformerscited by 31
  6. 2023On Uncertainty Calibration and Selective Generation in Probabilistic Neural Summarization: A Benchmark Studycited by 25
  7. 2024Do Not Design, Learn: A Trainable Scoring Function for Uncertainty Estimation in Generative LLMscited by 23
  8. 2024Enhancing Confidence Expression in Large Language Models Through Learning from Past Experiencecited by 21
  9. 2025Uncertainty-Aware Decoding with Minimum Bayes Riskcited by 11
  10. 2022Combining Data Generation and Active Learning for Low-Resource Question Answeringcited by 5
  11. 2023Logit-Based Ensemble Distribution Distillation for Robust Autoregressive Sequence Uncertaintiescited by 5
  12. 2024Enhancing Trust in Large Language Models via Uncertainty-Calibrated Fine-Tuningcited by 5
  13. 2025Improving the Calibration of Confidence Scores in Text Generation Using the Output Distribution's Characteristicscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026ConfidenceBench: Evaluating Confidence Calibration in Large Language Models
  2. 2025Verbalized Confidence Triggers Self-Verification: Emergent Behavior Without Explicit Reasoning Supervision
  3. 2026Rethinking Uncertainty Evaluation in Large Language Models
  4. 2024Confidence Under the Hood: An Investigation into the Confidence-Probability Alignment in Large Language Models
  5. 2026NOVA: NOise-aware Verbal Confidence CAlibration for Robust Large Language Models in RAG Systems
  6. 2026CALIBER: Calibrating Confidence Before and After Reasoning in Language Models
  7. 2026Know When You're Wrong: Aligning Confidence with Correctness for LLM Error Detection
  8. 2026Future Confidence Distillation in Large Language Models