# Learning to Evaluate Translation Beyond English: BLEURT Submissions to the WMT Metrics 2020 Shared Task

arXiv: https://arxiv.org/abs/2010.04297 · HTML: https://mlsanity.com/paper/2010.04297/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Oct 2020.

## Built on by (most cited first, 25 of 25)

- [Scaling Laws for Neural Machine Translation](https://mlsanity.com/paper/2109.07740/) (2021)
- [High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metrics](https://mlsanity.com/paper/2111.09388/) (2021)
- [Learning Compact Metrics for MT](https://mlsanity.com/paper/2110.06341/) (2021)
- [Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluation](https://mlsanity.com/paper/2202.07654/) (2022)
- [Epsilon Sampling Rocks: Investigating Sampling Strategies for Minimum Bayes Risk Decoding for Machine Translation](https://mlsanity.com/paper/2305.09860/) (2023)
- [Follow the Wisdom of the Crowd: Effective Text Generation via Minimum Bayes Risk Decoding](https://mlsanity.com/paper/2211.07634/) (2022)
- [Reward Gaming in Conditional Text Generation](https://mlsanity.com/paper/2211.08714/) (2022)
- [UniTE: Unified Translation Evaluation](https://mlsanity.com/paper/2204.13346/) (2022)
- [ACES: Translation Accuracy Challenge Sets for Evaluating Machine Translation Metrics](https://mlsanity.com/paper/2210.15615/) (2022)
- [INSTRUCTSCORE: Explainable Text Generation Evaluation with Finegrained Feedback](https://mlsanity.com/paper/2305.14282/) (2023)
- [SynthBio: A Case Study in Human-AI Collaborative Curation of Text Datasets](https://mlsanity.com/paper/2111.06467/) (2021)
- [Machine Translation Meta Evaluation through Translation Accuracy Challenge Sets](https://mlsanity.com/paper/2401.16313/) (2024)
- [Mitigating Metric Bias in Minimum Bayes Risk Decoding](https://mlsanity.com/paper/2411.03524/) (2024)
- [TransFool: An Adversarial Attack against Neural Machine Translation Models](https://mlsanity.com/paper/2302.00944/) (2023)
- [Has Machine Translation Evaluation Achieved Human Parity? The Human Reference and the Limits of Progress](https://mlsanity.com/paper/2506.19571/) (2025)
- [Disentangling Uncertainty in Machine Translation Evaluation](https://mlsanity.com/paper/2204.06546/) (2022)
- [TRUE: Re-evaluating Factual Consistency Evaluation](https://mlsanity.com/paper/2204.04991/) (2022)
- [Sentiment-Aware Measure (SAM) for Evaluating Sentiment Transfer by Machine Translation Systems](https://mlsanity.com/paper/2109.14895/) (2021)
- [A Classification-Guided Approach for Adversarial Attacks against Neural Machine Translation](https://mlsanity.com/paper/2308.15246/) (2023)
- [DIFFQG: Generating Questions to Summarize Factual Changes](https://mlsanity.com/paper/2303.00242/) (2023)
- [ACES: Translation Accuracy Challenge Sets at WMT 2023](https://mlsanity.com/paper/2311.01153/) (2023)
- [Learning Translation Quality Evaluation on Low Resource Languages from Large Language Models](https://mlsanity.com/paper/2302.03491/) (2023)
- [Joint Representations of Text and Knowledge Graphs for Retrieval and Evaluation](https://mlsanity.com/paper/2302.14785/) (2023)
- [PEAR: Pairwise Evaluation for Automatic Relative Scoring in Machine Translation](https://mlsanity.com/paper/2601.18006/) (2026)
- [Assessing and Improving Punctuation Robustness in English-Marathi Machine Translation](https://mlsanity.com/paper/2601.09725/) (2025)

## Abstract (arXiv)

The quality of machine translation systems has dramatically improved over the last decade, and as a result, evaluation has become an increasingly challenging problem. This paper describes our contribution to the WMT 2020 Metrics Shared Task, the main benchmark for automatic evaluation of translation. We make several submissions based on BLEURT, a previously published metric based on transfer learning. We extend the metric beyond English and evaluate it on 14 language pairs for which fine-tuning data is available, as well as 4 "zero-shot" language pairs, for which we have no labelled examples. Additionally, we focus on English to German and demonstrate how to combine BLEURT's predictions with those of YiSi and use alternative reference translations to enhance the performance. Empirical results show that the models achieve competitive results on the WMT Metrics 2019 Shared Task, indicating their promise for the 2020 edition.