# Mitigating Metric Bias in Minimum Bayes Risk Decoding arXiv: https://arxiv.org/abs/2411.03524 ยท HTML: https://mlsanity.com/paper/2411.03524/ ## Key facts - Cited by 16 papers in the map (data through 14 Oct 2026). - Builds on 26 of its 54 references that are in the map. - First posted on arXiv 5 Nov 2024. ## Builds on (most cited first, 26 of 26) - [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023) - [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017) - [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022) - [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018) - [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022) - [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020) - [Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation](https://mlsanity.com/paper/2104.14478/) (2021) - [xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detection](https://mlsanity.com/paper/2310.10482/) (2023) - [CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task](https://mlsanity.com/paper/2209.06243/) (2022) - [High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metrics](https://mlsanity.com/paper/2111.09388/) (2021) - [Scaling up COMETKIWI: Unbabel-IST 2023 Submission for the Quality Estimation Shared Task](https://mlsanity.com/paper/2309.11925/) (2023) - [Sampling-Based Approximations to Minimum Bayes Risk Decoding for Neural Machine Translation](https://mlsanity.com/paper/2108.04718/) (2021) - [Epsilon Sampling Rocks: Investigating Sampling Strategies for Minimum Bayes Risk Decoding for Machine Translation](https://mlsanity.com/paper/2305.09860/) (2023) - [Identifying Weaknesses in Machine Translation Metrics Through Minimum Bayes Risk Decoding: A Case Study for COMET](https://mlsanity.com/paper/2202.05148/) (2022) - [Faster Minimum Bayes Risk Decoding with Confidence-based Pruning](https://mlsanity.com/paper/2311.14919/) (2023) - [MBR and QE Finetuning: Training-time Distillation of the Best and Most Expensive Decoding Methods](https://mlsanity.com/paper/2309.10966/) (2023) - [Direct Preference Optimization for Neural Machine Translation with Minimum Bayes Risk Decoding](https://mlsanity.com/paper/2311.08380/) (2023) - [IndicMT Eval: A Dataset to Meta-Evaluate Machine Translation metrics for Indian Languages](https://mlsanity.com/paper/2212.10180/) (2022) - [Learning to Evaluate Translation Beyond English: BLEURT Submissions to the WMT Metrics 2020 Shared Task](https://mlsanity.com/paper/2010.04297/) (2020) - [Linear-time Minimum Bayes Risk Decoding with Reference Aggregation](https://mlsanity.com/paper/2402.04251/) (2024) - [Improving Neural Machine Translation with Conditional Sequence Generative Adversarial Nets](https://mlsanity.com/paper/1703.04887/) (2017) - [Hyperparameter-Free Approach for Faster Minimum Bayes Risk Decoding](https://mlsanity.com/paper/2401.02749/) (2024) - [Efficient Minimum Bayes Risk Decoding using Low-Rank Matrix Completion Algorithms](https://mlsanity.com/paper/2406.02832/) (2024) - [Centroid-Based Efficient Minimum Bayes Risk Decoding](https://mlsanity.com/paper/2402.11197/) (2024) - [Chasing COMET: Leveraging Minimum Bayes Risk Decoding for Self-Improving Machine Translation](https://mlsanity.com/paper/2405.11937/) (2024) - [Beyond Human-Only: Evaluating Human-Machine Collaboration for Collecting High-Quality Translation Data](https://mlsanity.com/paper/2410.11056/) (2024) ## Built on by (most cited first, 16 of 16) - [When Life Gives You Samples: The Benefits of Scaling up Inference Compute for Multilingual LLMs](https://mlsanity.com/paper/2506.20544/) (2025) - [LLMs Lost in Translation: M-ALERT uncovers Cross-Linguistic Safety Inconsistencies](https://mlsanity.com/paper/2412.15035/) (2024) - [Adding Chocolate to Mint: Mitigating Metric Interference in Machine Translation](https://mlsanity.com/paper/2503.08327/) (2025) - [Translate Smart, not Hard: Cascaded Translation Systems with Quality-Aware Deferral](https://mlsanity.com/paper/2502.12701/) (2025) - [Calibrating Translation Decoding with Quality Estimation on LLMs](https://mlsanity.com/paper/2504.19044/) (2025) - [Re-evaluating Minimum Bayes Risk Decoding for Automatic Speech Recognition](https://mlsanity.com/paper/2510.19471/) (2025) - [Preliminary Ranking of WMT25 General Machine Translation Systems](https://mlsanity.com/paper/2508.14909/) (2025) - [Minimum Bayes Risk Decoding for Error Span Detection in Reference-Free Automatic Machine Translation Evaluation](https://mlsanity.com/paper/2512.07540/) (2025) - [Uncertainty-Aware Generation and Decision-Making Under Ambiguity](https://mlsanity.com/paper/2606.30578/) (2026) - [Document-Level Text Generation with Minimum Bayes Risk Decoding using Optimal Transport](https://mlsanity.com/paper/2505.23078/) (2025) - [Multi-Hypothesis Distillation of Multilingual Neural Translation Models for Low-Resource Languages](https://mlsanity.com/paper/2507.21568/) (2025) - [Is Human Annotation Necessary? Iterative MBR Distillation for Error Span Detection in Machine Translation](https://mlsanity.com/paper/2603.12983/) (2026) - [Enhancing Factuality through Consensus and Consistency in Summarization Using Minimum Bayes Risk Decoding](https://mlsanity.com/paper/2605.29336/) (2026) - [Rewrite to Translate, Translate to Reward: Reinforcement Learning for Source Rewriting in Machine Translation](https://mlsanity.com/paper/2606.08011/) (2026) - [Overfitting Mitigation via Singular Value Decomposition in Minimum Bayes Risk Decoding](https://mlsanity.com/paper/2609.01135/) (2026) - [From Sampled Evidence to Radiology Reports: Objectives, Constraints, and Realization](https://mlsanity.com/paper/2610.17318/) (2026) ## Abstract (arXiv) While Minimum Bayes Risk (MBR) decoding using metrics such as COMET or MetricX has outperformed traditional decoding methods such as greedy or beam search, it introduces a challenge we refer to as metric bias. As MBR decoding aims to produce translations that score highly according to a specific utility metric, this very process makes it impossible to use the same metric for both decoding and evaluation, as improvements might simply be due to reward hacking rather than reflecting real quality improvements. In this work we find that compared to human ratings, neural metrics not only overestimate the quality of MBR decoding when the same metric is used as the utility metric, but they also overestimate the quality of MBR/QE decoding with other neural utility metrics as well. We also show that the metric bias issue can be mitigated by using an ensemble of utility metrics during MBR decoding: human evaluations show that MBR decoding using an ensemble of utility metrics outperforms a single utility metric.