# Unbabel's Participation in the WMT20 Metrics Shared Task

arXiv: https://arxiv.org/abs/2010.15535 · HTML: https://mlsanity.com/paper/2010.15535/

## Key facts

- Cited by 56 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Oct 2020.

## Built on by (most cited first, 30 of 56)

- [Learning Compact Metrics for MT](https://mlsanity.com/paper/2110.06341/) (2021)
- [Detecting and Mitigating Hallucinations in Machine Translation: Model Internal Workings Alone Do Well, Sentence Similarity Even Better](https://mlsanity.com/paper/2212.08597/) (2022)
- [Looking for a Needle in a Haystack: A Comprehensive Study of Hallucinations in Neural Machine Translation](https://mlsanity.com/paper/2208.05309/) (2022)
- [Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation](https://mlsanity.com/paper/2305.00955/) (2023)
- [Quality-Aware Decoding for Neural Machine Translation](https://mlsanity.com/paper/2205.00978/) (2022)
- [Understanding and Detecting Hallucinations in Neural Machine Translation via Model Introspection](https://mlsanity.com/paper/2301.07779/) (2023)
- [Identifying Weaknesses in Machine Translation Metrics Through Minimum Bayes Risk Decoding: A Case Study for COMET](https://mlsanity.com/paper/2202.05148/) (2022)
- [Uncertainty-Aware Machine Translation Evaluation](https://mlsanity.com/paper/2109.06352/) (2021)
- [MBR and QE Finetuning: Training-time Distillation of the Best and Most Expensive Decoding Methods](https://mlsanity.com/paper/2309.10966/) (2023)
- [Regularized Best-of-N Sampling with Minimum Bayes Risk Objective for Language Model Alignment](https://mlsanity.com/paper/2404.01054/) (2024)
- [Simple LLM Prompting is State-of-the-Art for Robust and Multilingual Dialogue Evaluation](https://mlsanity.com/paper/2308.16797/) (2023)
- [A Tale of Pronouns: Interpretability Informs Gender Bias Mitigation for Fairer Instruction-Tuned Machine Translation](https://mlsanity.com/paper/2310.12127/) (2023)
- [Aligning Neural Machine Translation Models: Human Feedback in Training and Inference](https://mlsanity.com/paper/2311.09132/) (2023)
- [MENLI: Robust Evaluation Metrics from Natural Language Inference](https://mlsanity.com/paper/2208.07316/) (2022)
- [Hyperparameter-Free Approach for Faster Minimum Bayes Risk Decoding](https://mlsanity.com/paper/2401.02749/) (2024)
- [MT-Ranker: Reference-free machine translation evaluation by inter-system ranking](https://mlsanity.com/paper/2401.17099/) (2024)
- [Compositional Translation: A Novel LLM-based Approach for Low-resource Machine Translation](https://mlsanity.com/paper/2503.04554/) (2025)
- [Optimal Transport for Unsupervised Hallucination Detection in Neural Machine Translation](https://mlsanity.com/paper/2212.09631/) (2022)
- [CTQScorer: Combining Multiple Features for In-context Example Selection for Machine Translation](https://mlsanity.com/paper/2305.14105/) (2023)
- [Poor Man's Quality Estimation: Predicting Reference-Based MT Metrics Without the Reference](https://mlsanity.com/paper/2301.09008/) (2023)
- [SLIDE: Reference-free Evaluation for Machine Translation using a Sliding Document Window](https://mlsanity.com/paper/2309.08832/) (2023)
- [Crossing the Threshold: Idiomatic Machine Translation through Retrieval Augmentation and Loss Weighting](https://mlsanity.com/paper/2310.07081/) (2023)
- [Model-Based Minimum Bayes Risk Decoding for Text Generation](https://mlsanity.com/paper/2311.05263/) (2023)
- [A Call for Clarity in Beam Search: How It Works and When It Stops](https://mlsanity.com/paper/2204.05424/) (2022)
- [COMET-QE and Active Learning for Low-Resource Machine Translation](https://mlsanity.com/paper/2210.15696/) (2022)
- [Critical Learning Periods: Leveraging Early Training Dynamics for Efficient Data Pruning](https://mlsanity.com/paper/2405.19462/) (2024)
- [Pitfalls and Outlooks in Using COMET](https://mlsanity.com/paper/2408.15366/) (2024)
- [Watching the Watchers: Exposing Gender Disparities in Machine Translation Quality Estimation](https://mlsanity.com/paper/2410.10995/) (2024)
- [Has Machine Translation Evaluation Achieved Human Parity? The Human Reference and the Limits of Progress](https://mlsanity.com/paper/2506.19571/) (2025)
- [Disentangling Uncertainty in Machine Translation Evaluation](https://mlsanity.com/paper/2204.06546/) (2022)

## Abstract (arXiv)

We present the contribution of the Unbabel team to the WMT 2020 Shared Task on Metrics. We intend to participate on the segment-level, document-level and system-level tracks on all language pairs, as well as the 'QE as a Metric' track. Accordingly, we illustrate results of our models in these tracks with reference to test sets from the previous year. Our submissions build upon the recently proposed COMET framework: We train several estimator models to regress on different human-generated quality scores and a novel ranking model trained on relative ranks obtained from Direct Assessments. We also propose a simple technique for converting segment-level predictions into a document-level score. Overall, our systems achieve strong results for all language pairs on previous test sets and in many cases set a new state-of-the-art.