mlsanityData to 7 Oct 2026

Are Generative Models Underconfident? Better Quality Estimation with Boosted Model Probability

arXiv 2502.11115 ↗First posted 16 Feb 2025cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Quality Estimation (QE) is estimating quality of the model output during inference when the ground truth is not available. Deriving output quality from the models' output probability is the most trivial and low-effort way. However, we show that the output probability of text-generation models can appear underconfident. At each output step, there can be multiple correct options, making the probability distribution spread out more. Thus, lower probability does not necessarily mean lower output quality. Due to this observation, we propose a QE approach called BoostedProb, which boosts the model's confidence in cases where there are multiple viable output options. With no increase in complexity, BoostedProb is notably better than raw model probability in different settings, achieving on average +0.194 improvement in Pearson correlation to ground-truth quality. It also comes close to or outperforms more costly approaches like supervised or ensemble-based QE in certain settings.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 41 of its 73 references that are in the map. First posted on arXiv 16 Feb 2025.

Builds on

41 of 73 references.

References · 41

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  7. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  8. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  9. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  10. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  11. 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,472
  12. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
29 more
  1. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  2. 2018Hierarchical Neural Story Generationcited by 1,182
  3. 2014Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Imagescited by 774
  4. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  5. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
  6. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
  7. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 322
  8. 2023xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detectioncited by 260
  9. 2020Unsupervised Quality Estimation for Neural Machine Translationcited by 210
  10. 2022CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Taskcited by 205
  11. 2024Tower: An Open Multilingual Large Language Model for Translation-Related Taskscited by 205
  12. 2022Out-of-Distribution Detection and Selective Generation for Conditional Language Modelscited by 193
  13. 2023LM vs LM: Detecting Factual Errors via Cross Examinationcited by 155
  14. 2024Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputscited by 148
  15. 2023LM-Polygraph: Uncertainty Estimation for Language Modelscited by 130
  16. 2018Analyzing Uncertainty in Neural Machine Translationcited by 126
  17. 2022Truncation Sampling as Language Model Desmoothingcited by 121
  18. 2020Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasingcited by 114
  19. 2024A New Massive Multilingual Dataset for High-Performance Language Technologiescited by 56
  20. 2021DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoderscited by 50
  21. 2020Confidence Estimation for Attention-based Sequence-to-sequence Models for Speech Recognitioncited by 35
  22. 2024Fine-Tuned Machine Translation Metrics Struggle in Unseen Domainscited by 21
  23. 2023There's no Data Like Better Data: Using QE Metrics for MT Data Filteringcited by 19
  24. 2021Quality Estimation without Human-labeled Datacited by 15
  25. 2024SciEx: Benchmarking Large Language Models on Scientific Exams with Human Expert Grading and Automatic Gradingcited by 14
  26. 2023Poor Man's Quality Estimation: Predicting Reference-Based MT Metrics Without the Referencecited by 11
  27. 2023Perturbation-based QE: An Explainable, Unsupervised Word-level Quality Estimation Method for Blackbox Machine Translationcited by 6
  28. 2023An Empirical Study of Pre-trained Model Selection for Out-of-Distribution Generalization and Calibrationcited by 5
  29. 2024Quality Estimation with k-nearest Neighbors and Automatic Evaluation for Model-specific Quality Estimationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Calibration Is Not Enough: Evaluating Confidence Estimation Under Language Variationscited by 3
  2. 2026Speaking in Self-Assessing Tongues: On the Verbalized Confidence of LLMs in Machine Translationcited by 1
  3. 2026Sigmoid Head for Quality Estimation under Language Ambiguitycited by 0
  4. 2026On the Rejection Criterion for Proxy-based Test-time Alignmentcited by 0