# MBR and QE Finetuning: Training-time Distillation of the Best and Most Expensive Decoding Methods

arXiv: https://arxiv.org/abs/2309.10966 · HTML: https://mlsanity.com/paper/2309.10966/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 37 references that are in the map.
- First posted on arXiv 19 Sep 2023.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [PaLM 2 Technical Report](https://mlsanity.com/paper/2305.10403/) (2023)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [Reinforced Self-Training (ReST) for Language Modeling](https://mlsanity.com/paper/2308.08998/) (2023)
- [Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation](https://mlsanity.com/paper/2104.14478/) (2021)
- [Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translation](https://mlsanity.com/paper/2005.10283/) (2020)
- [Generalization in Generation: A closer look at Exposure Bias](https://mlsanity.com/paper/1910.00292/) (2019)
- [Truncation Sampling as Language Model Desmoothing](https://mlsanity.com/paper/2210.15191/) (2022)
- [Minimum Risk Training for Neural Machine Translation](https://mlsanity.com/paper/1512.02433/) (2015)
- [High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metrics](https://mlsanity.com/paper/2111.09388/) (2021)
- [Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling](https://mlsanity.com/paper/1902.08295/) (2019)
- [Sampling-Based Approximations to Minimum Bayes Risk Decoding for Neural Machine Translation](https://mlsanity.com/paper/2108.04718/) (2021)
- [Epsilon Sampling Rocks: Investigating Sampling Strategies for Minimum Bayes Risk Decoding for Machine Translation](https://mlsanity.com/paper/2305.09860/) (2023)
- [Unbabel's Participation in the WMT20 Metrics Shared Task](https://mlsanity.com/paper/2010.15535/) (2020)
- [Quality-Aware Decoding for Neural Machine Translation](https://mlsanity.com/paper/2205.00978/) (2022)
- [Denoising Neural Machine Translation Training with Trusted Data and Online Data Selection](https://mlsanity.com/paper/1809.00068/) (2018)
- [Identifying Weaknesses in Machine Translation Metrics Through Minimum Bayes Risk Decoding: A Case Study for COMET](https://mlsanity.com/paper/2202.05148/) (2022)
- [Fast Domain Adaptation for Neural Machine Translation](https://mlsanity.com/paper/1612.06897/) (2016)

## Built on by (most cited first, 30 of 33)

- [Improving Machine Translation with Human Feedback: An Exploration of Quality Estimation as a Reward Model](https://mlsanity.com/paper/2401.12873/) (2024)
- [Direct Preference Optimization for Neural Machine Translation with Minimum Bayes Risk Decoding](https://mlsanity.com/paper/2311.08380/) (2023)
- [Regularized Best-of-N Sampling with Minimum Bayes Risk Objective for Language Model Alignment](https://mlsanity.com/paper/2404.01054/) (2024)
- [Linear-time Minimum Bayes Risk Decoding with Reference Aggregation](https://mlsanity.com/paper/2402.04251/) (2024)
- [Better Instruction-Following Through Minimum Bayes Risk](https://mlsanity.com/paper/2410.02902/) (2024)
- [Mitigating Metric Bias in Minimum Bayes Risk Decoding](https://mlsanity.com/paper/2411.03524/) (2024)
- [Hyperparameter-Free Approach for Faster Minimum Bayes Risk Decoding](https://mlsanity.com/paper/2401.02749/) (2024)
- [Regression-aware Inference with LLMs](https://mlsanity.com/paper/2403.04182/) (2024)
- [AdaCQR: Enhancing Query Reformulation for Conversational Search via Sparse and Dense Retrieval Alignment](https://mlsanity.com/paper/2407.01965/) (2024)
- [Don't Rank, Combine! Combining Machine Translation Hypotheses Using Quality Estimation](https://mlsanity.com/paper/2401.06688/) (2024)
- [Efficient Minimum Bayes Risk Decoding using Low-Rank Matrix Completion Algorithms](https://mlsanity.com/paper/2406.02832/) (2024)
- [Model-Based Minimum Bayes Risk Decoding for Text Generation](https://mlsanity.com/paper/2311.05263/) (2023)
- [Uncertainty-Aware Decoding with Minimum Bayes Risk](https://mlsanity.com/paper/2503.05318/) (2025)
- [Introducing the NewsPaLM MBR and QE Dataset: LLM-Generated High-Quality Parallel Data Outperforms Traditional Web-Crawled Data](https://mlsanity.com/paper/2408.06537/) (2024)
- [Pitfalls and Outlooks in Using COMET](https://mlsanity.com/paper/2408.15366/) (2024)
- [Beyond Correlation: Interpretable Evaluation of Machine Translation Metrics](https://mlsanity.com/paper/2410.05183/) (2024)
- [Don't Throw Away Data: Better Sequence Knowledge Distillation](https://mlsanity.com/paper/2407.10456/) (2024)
- [Chasing COMET: Leveraging Minimum Bayes Risk Decoding for Self-Improving Machine Translation](https://mlsanity.com/paper/2405.11937/) (2024)
- [Choose the Final Translation from NMT and LLM hypotheses Using MBR Decoding: HW-TSC's Submission to the WMT24 General MT Shared Task](https://mlsanity.com/paper/2409.14800/) (2024)
- [Is Preference Alignment Always the Best Option to Enhance LLM-Based Translation? An Empirical Analysis](https://mlsanity.com/paper/2409.20059/) (2024)
- [Mufu: Multilingual Fused Learning for Low-Resource Translation with LLM](https://mlsanity.com/paper/2409.13949/) (2024)
- [Learning from others' mistakes: Finetuning machine translation models with span-level error annotations](https://mlsanity.com/paper/2410.16509/) (2024)
- [CRPO: Confidence-Reward Driven Preference Optimization for Machine Translation](https://mlsanity.com/paper/2501.13927/) (2025)
- [On the True Distribution Approximation of Minimum Bayes-Risk Decoding](https://mlsanity.com/paper/2404.00752/) (2024)
- [A Bayesian Optimization Approach to Machine Translation Reranking](https://mlsanity.com/paper/2411.09694/) (2024)
- [A Context-aware Framework for Translation-mediated Conversations](https://mlsanity.com/paper/2412.04205/) (2024)
- [Early-Exit and Instant Confidence Translation Quality Estimation](https://mlsanity.com/paper/2502.14429/) (2025)
- [KD4MT: A Survey of Knowledge Distillation for Machine Translation](https://mlsanity.com/paper/2602.15845/) (2026)
- [Preliminary Ranking of WMT25 General Machine Translation Systems](https://mlsanity.com/paper/2508.14909/) (2025)
- [Consensus Group Relative Policy Optimization for Text Generation](https://mlsanity.com/paper/2602.03102/) (2026)

## Abstract (arXiv)

Recent research in decoding methods for Natural Language Generation (NLG) tasks has shown that MAP decoding is not optimal, because model probabilities do not always align with human preferences. Stronger decoding methods, including Quality Estimation (QE) reranking and Minimum Bayes' Risk (MBR) decoding, have since been proposed to mitigate the model-perplexity-vs-quality mismatch. While these decoding methods achieve state-of-the-art performance, they are prohibitively expensive to compute. In this work, we propose MBR finetuning and QE finetuning which distill the quality gains from these decoding methods at training time, while using an efficient decoding algorithm at inference time. Using the canonical NLG task of Neural Machine Translation (NMT), we show that even with self-training, these finetuning methods significantly outperform the base model. Moreover, when using an external LLM as a teacher model, these finetuning methods outperform finetuning on human-generated references. These findings suggest new ways to leverage monolingual data to achieve improvements in model quality that are on par with, or even exceed, improvements from human-curated data, while maintaining maximum efficiency during decoding.