mlsanityData to 7 Oct 2026

High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metrics

arXiv 2111.09388 ↗First posted 17 Nov 2021cs.CL cs.AI cs.LGcited by 104

Not on the map. Why?

Abstract (arXiv)

In Neural Machine Translation, it is typically assumed that the sentence with the highest estimated probability should also be the translation with the highest quality as measured by humans. In this work, we question this assumption and show that model estimates and translation quality only vaguely correlate. We apply Minimum Bayes Risk (MBR) decoding on unbiased samples to optimize diverse automated metrics of translation quality as an alternative inference strategy to beam search. Instead of targeting the hypotheses with the highest model probability, MBR decoding extracts the hypotheses with the highest estimated quality. Our experiments show that the combination of a neural translation model with a neural reference-based metric, BLEURT, results in significant improvement in human evaluations. This improvement is obtained with translations different from classical beam-search output: these translations have much lower model likelihood and are less favored by surface metrics like BLEU.

Key facts. Cited by 104 papers in the map (data through 7 Oct 2026). Builds on 26 of its 54 references that are in the map. First posted on arXiv 17 Nov 2021.

Builds on

26 of 54 references.

References · 26

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  4. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  5. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  7. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  8. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  9. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  10. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  11. 2017Convolutional Sequence to Sequence Learningcited by 550
  12. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 333
14 more
  1. 2018Scaling Neural Machine Translationcited by 204
  2. 2016An Actor-Critic Algorithm for Sequence Predictioncited by 158
  3. 2020Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translationcited by 137
  4. 2020BLEU might be Guilty but References are not Innocentcited by 109
  5. 2020Rethinking embedding coupling in pre-trained language modelscited by 103
  6. 2019Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modelingcited by 86
  7. 2021A Statistical Analysis of Summarization Evaluation Metrics using Resampling Methodscited by 71
  8. 2021Facebook AI WMT21 News Translation Task Submissioncited by 57
  9. 2021Understanding the Properties of Minimum Bayes Risk Decoding in Neural Machine Translationcited by 56
  10. 2017Classical Structured Prediction Losses for Sequence to Sequence Learningcited by 55
  11. 2021Machine Translation Decoding beyond Beam Searchcited by 55
  12. 2018Denoising Neural Machine Translation Training with Trusted Data and Online Data Selectioncited by 43
  13. 2020Learning to Evaluate Translation Beyond English: BLEURT Submissions to the WMT Metrics 2020 Shared Taskcited by 25
  14. 2020Reassessing Claims of Human Parity and Super-Human Performance in Machine Translation at WMT 2019cited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 9; 2023: 26; 2024: 37; 2025: 20; 2026: 12.

104 papers.

Sort:

By area:

  1. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
  2. 2024Tower: An Open Multilingual Large Language Model for Translation-Related Taskscited by 205
  3. 2023DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedbackcited by 99
  4. 2023The unreasonable effectiveness of few-shot learning for machine translationcited by 91
  5. 2023Ties Matter: Meta-Evaluating Modern Metrics with Pairwise Accuracy and Tie Calibrationcited by 71
  6. 2022Looking for a Needle in a Haystack: A Comprehensive Study of Hallucinations in Neural Machine Translationcited by 60
  7. 2023Epsilon Sampling Rocks: Investigating Sampling Strategies for Minimum Bayes Risk Decoding for Machine Translationcited by 57
  8. 2023Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generationcited by 55
  9. 2022Follow the Wisdom of the Crowd: Effective Text Generation via Minimum Bayes Risk Decodingcited by 54
  10. 2022LENS: A Learnable Evaluation Metric for Text Simplificationcited by 54
  11. 2022On the Limitations of Reference-Free Evaluations of Generated Textcited by 52
  12. 2023It's MBR All the Way Down: Modern Generation Techniques Through the Lens of Minimum Bayes Riskcited by 49
38 more
  1. 2023Mitigating Hallucinations and Off-target Machine Translation with Source-Contrastive and Language-Contrastive Decodingcited by 42
  2. 2022Identifying Weaknesses in Machine Translation Metrics Through Minimum Bayes Risk Decoding: A Case Study for COMETcited by 41
  3. 2024Navigating the Metrics Maze: Reconciling Score Magnitudes and Accuraciescited by 40
  4. 2023LLMRefine: Pinpointing and Refining Large Language Models via Fine-Grained Actionable Feedbackcited by 39
  5. 2023Faster Minimum Bayes Risk Decoding with Confidence-based Pruningcited by 34
  6. 2023MBR and QE Finetuning: Training-time Distillation of the Best and Most Expensive Decoding Methodscited by 33
  7. 2023Direct Preference Optimization for Neural Machine Translation with Minimum Bayes Risk Decodingcited by 28
  8. 2024Blending Is All You Need: Cheaper, Better Alternative to Trillion-Parameters LLMcited by 26
  9. 2023BLEURT Has Universal Translations: An Analysis of Automatic Metrics by Minimum Risk Trainingcited by 24
  10. 2024Regularized Best-of-N Sampling with Minimum Bayes Risk Objective for Language Model Alignmentcited by 24
  11. 2024Linear-time Minimum Bayes Risk Decoding with Reference Aggregationcited by 21
  12. 2023Aligning Neural Machine Translation Models: Human Feedback in Training and Inferencecited by 19
  13. 2024Better Instruction-Following Through Minimum Bayes Riskcited by 19
  14. 2023Small Language Models Improve Giants by Rewriting Their Outputscited by 18
  15. 2023An Empirical Study of Translation Hypothesis Ensembling with Large Language Modelscited by 16
  16. 2024Can Automatic Metrics Assess High-Quality Translations?cited by 16
  17. 2024QUEST: Quality-Aware Metropolis-Hastings Sampling for Machine Translationcited by 16
  18. 2024Hyperparameter-Free Approach for Faster Minimum Bayes Risk Decodingcited by 15
  19. 2024Mitigating Metric Bias in Minimum Bayes Risk Decodingcited by 15
  20. 2025Control the Temperature: Selective Sampling for Diverse and High-Quality LLM Outputscited by 15
  21. 2024Don't Rank, Combine! Combining Machine Translation Hypotheses Using Quality Estimationcited by 13
  22. 2024Efficient Minimum Bayes Risk Decoding using Low-Rank Matrix Completion Algorithmscited by 13
  23. 2024Improving Minimum Bayes Risk Decoding with Multi-Promptcited by 13
  24. 2024mbrs: A Library for Minimum Bayes Risk Decodingcited by 13
  25. 2023Binarized Neural Machine Translationcited by 12
  26. 2024Generating Diverse and High-Quality Texts by Minimum Bayes Risk Decodingcited by 11
  27. 2023R-U-SURE? Uncertainty-Aware Code Suggestions By Maximizing Utility Across Random User Intentscited by 10
  28. 2023Model-Based Minimum Bayes Risk Decoding for Text Generationcited by 10
  29. 2024Introducing the NewsPaLM MBR and QE Dataset: LLM-Generated High-Quality Parallel Data Outperforms Traditional Web-Crawled Datacited by 10
  30. 2024Pitfalls and Outlooks in Using COMETcited by 10
  31. 2024Beyond Correlation: Interpretable Evaluation of Machine Translation Metricscited by 10
  32. 2024Watching the Watchers: Exposing Gender Disparities in Machine Translation Quality Estimationcited by 10
  33. 2025Uncertainty-Aware Decoding with Minimum Bayes Riskcited by 10
  34. 2022SESCORE2: Learning Text Generation Evaluation via Synthesizing Realistic Mistakescited by 9
  35. 2023Quality-Aware Translation Models: Efficient Generation and Quality Estimation in a Single Modelcited by 9
  36. 2024Don't Throw Away Data: Better Sequence Knowledge Distillationcited by 9
  37. 2023CUED at ProbSum 2023: Hierarchical Ensemble of Summarization Modelscited by 8
  38. 2023Language Model Decoding as Direct Metrics Optimizationcited by 8

50 most cited of 104 All 104 (JSON)