mlsanityData to 7 Oct 2026

Identifying Weaknesses in Machine Translation Metrics Through Minimum Bayes Risk Decoding: A Case Study for COMET

arXiv 2202.05148 ↗First posted 10 Feb 2022cs.CLcited by 41

Not on the map. Why?

Abstract (arXiv)

Neural metrics have achieved impressive correlation with human judgements in the evaluation of machine translation systems, but before we can safely optimise towards such metrics, we should be aware of (and ideally eliminate) biases toward bad translations that receive high scores. Our experiments show that sample-based Minimum Bayes Risk decoding can be used to explore and quantify such weaknesses. When applying this strategy to COMET for en-de and de-en, we find that COMET models are not sensitive enough to discrepancies in numbers and named entities. We further show that these biases are hard to fully remove by simply training on additional synthetic data and release our code and data for facilitating further experiments.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Builds on 22 of its 55 references that are in the map. First posted on arXiv 10 Feb 2022.

Builds on

22 of 55 references.

References · 22

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  4. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  5. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  6. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  7. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  8. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  9. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  10. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 333
  11. 2016Using the Output Embedding to Improve Language Modelscited by 266
  12. 2018Marian: Fast Neural Machine Translation in C++cited by 202
10 more
  1. 2019BPE-Dropout: Simple and Effective Subword Regularizationcited by 189
  2. 2021To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translationcited by 161
  3. 2020Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translationcited by 137
  4. 2015Minimum Risk Training for Neural Machine Translationcited by 114
  5. 2020Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-trained Language Modelscited by 110
  6. 2021High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metricscited by 104
  7. 2020Unbabel's Participation in the WMT20 Metrics Shared Taskcited by 56
  8. 2021Understanding the Properties of Minimum Bayes Risk Decoding in Neural Machine Translationcited by 56
  9. 2017Nematus: a Toolkit for Neural Machine Translationcited by 16
  10. 2021As Easy as 1, 2, 3: Behavioural Testing of NMT Systems for Numerical Translationcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 11; 2023: 12; 2024: 15; 2025: 2; 2026: 1.

41 papers.

Sort:

By area:

  1. 2023xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detectioncited by 260
  2. 2023Uncertainty in Natural Language Generation: From Theory to Applicationscited by 61
  3. 2022Looking for a Needle in a Haystack: A Comprehensive Study of Hallucinations in Neural Machine Translationcited by 60
  4. 2023Epsilon Sampling Rocks: Investigating Sampling Strategies for Minimum Bayes Risk Decoding for Machine Translationcited by 57
  5. 2023Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generationcited by 55
  6. 2022LENS: A Learnable Evaluation Metric for Text Simplificationcited by 54
  7. 2022Quality-Aware Decoding for Neural Machine Translationcited by 49
  8. 2023It's MBR All the Way Down: Modern Generation Techniques Through the Lens of Minimum Bayes Riskcited by 49
  9. 2022DEMETR: Diagnosing Evaluation Metrics for Translationcited by 36
  10. 2023MBR and QE Finetuning: Training-time Distillation of the Best and Most Expensive Decoding Methodscited by 33
  11. 2022Reward Gaming in Conditional Text Generationcited by 29
  12. 2022ACES: Translation Accuracy Challenge Sets for Evaluating Machine Translation Metricscited by 25
29 more
  1. 2023BLEURT Has Universal Translations: An Analysis of Automatic Metrics by Minimum Risk Trainingcited by 24
  2. 2024Linear-time Minimum Bayes Risk Decoding with Reference Aggregationcited by 21
  3. 2024Machine Translation Meta Evaluation through Translation Accuracy Challenge Setscited by 18
  4. 2024QUEST: Quality-Aware Metropolis-Hastings Sampling for Machine Translationcited by 16
  5. 2023The Inside Story: Towards Better Understanding of Machine Translation Neural Evaluation Metricscited by 15
  6. 2024Hyperparameter-Free Approach for Faster Minimum Bayes Risk Decodingcited by 15
  7. 2024Mitigating Metric Bias in Minimum Bayes Risk Decodingcited by 15
  8. 2022Extrinsic Evaluation of Machine Translation Metricscited by 14
  9. 2024Efficient Minimum Bayes Risk Decoding using Low-Rank Matrix Completion Algorithmscited by 13
  10. 2024Improving Minimum Bayes Risk Decoding with Multi-Promptcited by 13
  11. 2024Pitfalls and Outlooks in Using COMETcited by 10
  12. 2024Watching the Watchers: Exposing Gender Disparities in Machine Translation Quality Estimationcited by 10
  13. 2022Disentangling Uncertainty in Machine Translation Evaluationcited by 9
  14. 2023Quality-Aware Translation Models: Efficient Generation and Quality Estimation in a Single Modelcited by 9
  15. 2022Who Are We Talking About? Handling Person Names in Speech Translationcited by 8
  16. 2024Diversity Explains Inference Scaling Laws: Through a Case Study of Minimum Bayes Risk Decodingcited by 8
  17. 2024Chasing COMET: Leveraging Minimum Bayes Risk Decoding for Self-Improving Machine Translationcited by 6
  18. 2022Adam Mickiewicz University at WMT 2022: NER-Assisted and Quality-Aware Neural Machine Translationcited by 5
  19. 2023BLEU Meets COMET: Combining Lexical and Neural Metrics Towards Robust Machine Translation Evaluationcited by 5
  20. 2023Breeding Machine Translations: Evolutionary approach to survive and thrive in the world of automated evaluationcited by 5
  21. 2024Is Preference Alignment Always the Best Option to Enhance LLM-Based Translation? An Empirical Analysiscited by 4
  22. 2025Specification-Aware Machine Translation and Evaluation for Purpose Alignmentcited by 4
  23. 2022CUNI Submission in WMT22 General Taskcited by 2
  24. 2024On the True Distribution Approximation of Minimum Bayes-Risk Decodingcited by 2
  25. 2024Favi-Score: A Measure for Favoritism in Automated Preference Ratings for Generative AI Evaluationcited by 2
  26. 2025Context-Enhanced Contrastive Search for Improved LLM Text Generationcited by 2
  27. 2023Trained MT Metrics Learn to Cope with Machine-translated Referencescited by 0
  28. 2024Thesis proposal: Are We Losing Textual Diversity to Natural Language Processing?cited by 0
  29. 2026In the Blind: Building Pseudo-References for MT Evaluationcited by 0