# Confident in a Confidence Score: Investigating the Sensitivity of Confidence Scores to Supervised Fine-Tuning

arXiv: https://arxiv.org/abs/2604.08974 · HTML: https://mlsanity.com/paper/2604.08974/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 40 references that are in the map.
- First posted on arXiv 10 Apr 2026.

## Builds on (most cited first, 25 of 25)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback](https://mlsanity.com/paper/2305.14975/) (2023)
- [Teaching Models to Express Their Uncertainty in Words](https://mlsanity.com/paper/2205.14334/) (2022)
- [The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation](https://mlsanity.com/paper/2106.03193/) (2021)
- [Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness](https://mlsanity.com/paper/2006.10108/) (2020)
- [Selective Question Answering under Domain Shift](https://mlsanity.com/paper/2006.09462/) (2020)
- [Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similarities](https://mlsanity.com/paper/2405.20003/) (2024)
- [Ensemble Distribution Distillation](https://mlsanity.com/paper/1905.00076/) (2019)
- [Correcting Length Bias in Neural Machine Translation](https://mlsanity.com/paper/1808.10006/) (2018)
- [Taming Overconfidence in LLMs: Reward Calibration in RLHF](https://mlsanity.com/paper/2410.09724/) (2024)
- [Calibration of Encoder Decoder Models for Neural Machine Translation](https://mlsanity.com/paper/1903.00802/) (2019)
- [Wat zei je? Detecting Out-of-Distribution Translations with Variational Transformers](https://mlsanity.com/paper/2006.08344/) (2020)
- [On Uncertainty Calibration and Selective Generation in Probabilistic Neural Summarization: A Benchmark Study](https://mlsanity.com/paper/2304.08653/) (2023)
- [Do Not Design, Learn: A Trainable Scoring Function for Uncertainty Estimation in Generative LLMs](https://mlsanity.com/paper/2406.11278/) (2024)
- [Enhancing Confidence Expression in Large Language Models Through Learning from Past Experience](https://mlsanity.com/paper/2404.10315/) (2024)
- [Uncertainty-Aware Decoding with Minimum Bayes Risk](https://mlsanity.com/paper/2503.05318/) (2025)
- [Combining Data Generation and Active Learning for Low-Resource Question Answering](https://mlsanity.com/paper/2211.14880/) (2022)
- [Logit-Based Ensemble Distribution Distillation for Robust Autoregressive Sequence Uncertainties](https://mlsanity.com/paper/2305.10384/) (2023)
- [Enhancing Trust in Large Language Models via Uncertainty-Calibrated Fine-Tuning](https://mlsanity.com/paper/2412.02904/) (2024)
- [Improving the Calibration of Confidence Scores in Text Generation Using the Output Distribution's Characteristics](https://mlsanity.com/paper/2506.00637/) (2025)

## Abstract (arXiv)

Uncertainty quantification techniques measure confidence in language model outputs to support critical applications like hallucination detection and selective prediction. While prior work has developed various confidence metrics and demonstrated their calibration for classification tasks or using verbalized confidence, the robustness of probability-based and self-consistency-based UQ metrics for natural language generation remains underexplored particularly under model adaptation. Since practitioners routinely apply supervised fine-tuning to adapt models to new tasks, a key question arises: do confidence metrics maintain their calibration when models are fine-tuned? We investigate this question across NLG tasks including translation, question answering, and mathematical reasoning. We find that calibration shifts substantially after SFT: across 216 configurations, it degrades in 112 cases and improves in 104, with confidence scores shifting due to factors beyond output quality, such as proximity to the training distribution. Degradation is therefore neither universal nor rare, and its direction cannot be anticipated from the pre-SFT model. Through a downstream task evaluation, we show that this miscalibration substantially reduces the practical utility of confidence scores for identifying correct answers. Our findings reveal that existing confidence metrics for NLG cannot be reliably deployed off-the-shelf after fine-tuning, highlighting the need for calibration-robust UQ methods under model adaptation.