mlsanityData to 7 Oct 2026

Uncertainty-Aware Machine Translation Evaluation

arXiv 2109.06352 ↗First posted 13 Sep 2021cs.CL cs.LGcited by 35

Not on the map. Why?

Abstract (arXiv)

Several neural-based metrics have been recently proposed to evaluate machine translation quality. However, all of them resort to point estimates, which provide limited information at segment level. This is made worse as they are trained on noisy, biased and scarce human judgements, often resulting in unreliable quality predictions. In this paper, we introduce uncertainty-aware MT evaluation and analyze the trustworthiness of the predicted quality. We combine the COMET framework with two uncertainty estimation methods, Monte Carlo dropout and deep ensembles, to obtain quality scores along with confidence intervals. We compare the performance of our uncertainty-aware MT evaluation methods across multiple language pairs from the QT21 dataset and the WMT20 metrics task, augmented with MQM annotations. We experiment with varying numbers of references and further discuss the usefulness of uncertainty-aware quality estimation (without references) to flag possibly critical translation mistakes.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Builds on 35 of its 77 references that are in the map. First posted on arXiv 13 Sep 2021.

Builds on

35 of 77 references.

References · 35

  1. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  2. 2017On Calibration of Modern Neural Networkscited by 4,860
  3. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  4. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  5. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
  6. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  7. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  8. 2019Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methodscited by 903
  9. 2018Evidential Deep Learning to Quantify Classification Uncertaintycited by 728
  10. 2018Predictive Uncertainty Estimation via Prior Networkscited by 555
  11. 2018Accurate Uncertainties for Deep Learning Using Calibrated Regressioncited by 417
  12. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 333
23 more
  1. 2020Calibration of Pre-trained Transformerscited by 307
  2. 2015Bayesian SegNet: Model Uncertainty in Deep Convolutional Encoder-Decoder Architectures for Scene Understandingcited by 218
  3. 2020Unsupervised Quality Estimation for Neural Machine Translationcited by 210
  4. 2020Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learningcited by 208
  5. 2018Understanding Measures of Uncertainty for Adversarial Example Detectioncited by 177
  6. 2020Posterior Network: Uncertainty Estimation without OOD Samples via Density-Based Pseudo-Countscited by 163
  7. 2018Analyzing Uncertainty in Neural Machine Translationcited by 126
  8. 2020Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasingcited by 114
  9. 2020BLEU might be Guilty but References are not Innocentcited by 109
  10. 2018Uncertainty in Neural Networks: Approximately Bayesian Ensemblingcited by 94
  11. 2018Bayesian Layers: A Module for Neural Network Uncertaintycited by 57
  12. 2020Unbabel's Participation in the WMT20 Metrics Shared Taskcited by 56
  13. 2019OpenKiwi: An Open Source Framework for Quality Estimationcited by 49
  14. 2020Predicting Performance for Natural Language Processing Taskscited by 49
  15. 2020Paraphrase Generation as Zero-Shot Multilingual Translation: Disentangling Semantic Similarity from Lexical and Syntactic Diversitycited by 35
  16. 2019Improving Back-Translation with Uncertainty-based Confidence Estimationcited by 34
  17. 2019Maximizing Overall Diversity for Improved Uncertainty Estimates in Deep Ensemblescited by 30
  18. 2021Towards More Fine-grained and Reliable NLP Performance Predictioncited by 26
  19. 2020Notes on the Behavior of MC Dropoutcited by 21
  20. 2020Reassessing Claims of Human Parity and Super-Human Performance in Machine Translation at WMT 2019cited by 20
  21. 2020TransQuest at WMT2020: Sentence-Level Direct Assessmentcited by 20
  22. 2017Modeling Source Syntax for Neural Machine Translationcited by 12
  23. 2016Exploring Prediction Uncertainty in Machine Translation Quality Estimationcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 12; 2024: 8; 2025: 10; 2026: 1.

35 papers.

Sort:

By area:

  1. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  2. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  3. 2023Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Modelscited by 147
  4. 2024Calibrating Large Language Models Using Their Generations Onlycited by 63
  5. 2023Uncertainty in Natural Language Generation: From Theory to Applicationscited by 61
  6. 2023Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generationcited by 55
  7. 2024Conformal Prediction for Natural Language Processing: A Surveycited by 52
  8. 2023What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variabilitycited by 51
  9. 2023Uncertainty in Natural Language Processing: Sources, Quantification, and Applicationscited by 44
  10. 2022Exploring Predictive Uncertainty and Calibration in NLP: A Study on the Impact of Method & Data Scarcitycited by 24
  11. 2024Non-Exchangeable Conformal Language Generation with Nearest Neighborscited by 21
  12. 2024Word-Sequence Entropy: Towards Uncertainty Estimation in Free-Form Medical Question Answering Applications and Beyondcited by 18
23 more
  1. 2025Uncertainty Quantification of Large Language Models through Multi-Dimensional Responsescited by 17
  2. 2024Interpreting Predictive Probabilities: Model Confidence or Human Label Variation?cited by 15
  3. 2023Poor Man's Quality Estimation: Predicting Reference-Based MT Metrics Without the Referencecited by 11
  4. 2023Simple Token-Level Confidence Improves Caption Correctnesscited by 11
  5. 2024Pitfalls and Outlooks in Using COMETcited by 10
  6. 2022Disentangling Uncertainty in Machine Translation Evaluationcited by 9
  7. 2023Evaluating Machine Translation Quality with Conformal Predictive Distributionscited by 9
  8. 2024Understanding Knowledge Drift in LLMs through Misinformationcited by 9
  9. 2024Integrative Decoding: Improve Factuality via Implicit Self-consistencycited by 8
  10. 2022Alibaba-Translate China's Submission for WMT 2022 Metrics Shared Taskcited by 7
  11. 2025Adaptive Elicitation of Latent Information Using Natural Languagecited by 7
  12. 2025Random-Set Large Language Modelscited by 7
  13. 2025Adding Chocolate to Mint: Mitigating Metric Interference in Machine Translationcited by 6
  14. 2023Ambiguity Meets Uncertainty: Investigating Uncertainty Estimation for Word Sense Disambiguationcited by 5
  15. 2023Conformalizing Machine Translation Evaluationcited by 5
  16. 2022Alibaba-Translate China's Submission for WMT 2022 Quality Estimation Shared Taskcited by 3
  17. 2025Early-Exit and Instant Confidence Translation Quality Estimationcited by 2
  18. 2025Uncertainty Quantification for Evaluating Machine Translation Biascited by 2
  19. 2025Efficient semantic uncertainty quantification in language models via diversity-steered samplingcited by 2
  20. 2025A statistically consistent measure of semantic uncertainty using Language Modelscited by 1
  21. 2025PIE: Performance Interval Estimation for Free-Form Generation Taskscited by 1
  22. 2025A Conformal Risk Control Framework for Granular Word Assessment and Uncertainty Calibration of CLIPScore Quality Estimatescited by 0
  23. 2026Looking under the Wrong Lamppost: On the Limitations of Automated Translation Quality Estimationcited by 0