# Quality and Quantity of Machine Translation References for Automatic Metrics

arXiv: https://arxiv.org/abs/2401.01283 · HTML: https://mlsanity.com/paper/2401.01283/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 53 references that are in the map.
- First posted on arXiv 2 Jan 2024.

## Builds on (most cited first, 16 of 16)

- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation](https://mlsanity.com/paper/2104.14478/) (2021)
- [To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translation](https://mlsanity.com/paper/2107.10821/) (2021)
- [Analyzing Uncertainty in Neural Machine Translation](https://mlsanity.com/paper/1803.00047/) (2018)
- [BLEU might be Guilty but References are not Innocent](https://mlsanity.com/paper/2004.06063/) (2020)
- [Embarrassingly Easy Document-Level MT Metrics: How to Convert Any Pretrained Metric Into a Document-Level Metric](https://mlsanity.com/paper/2209.13654/) (2022)
- [A Challenge Set Approach to Evaluating Machine Translation](https://mlsanity.com/paper/1704.07431/) (2017)
- [Dynamic Data Selection for Neural Machine Translation](https://mlsanity.com/paper/1708.00712/) (2017)
- [Simulated Multiple Reference Training Improves Low-Resource Machine Translation](https://mlsanity.com/paper/2004.14524/) (2020)
- [Multi-Reference Training with Pseudo-References for Neural Translation and Text Generation](https://mlsanity.com/paper/1808.09564/) (2018)
- [Neural Machine Translation Quality and Post-Editing Performance](https://mlsanity.com/paper/2109.05016/) (2021)
- [Onception: Active Learning with Expert Advice for Real World Machine Translation](https://mlsanity.com/paper/2203.04507/) (2022)
- [Evaluating Optimal Reference Translations](https://mlsanity.com/paper/2311.16787/) (2023)

## Built on by (most cited first, 7 of 7)

- [Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontiers](https://mlsanity.com/paper/2404.04925/) (2024)
- [AI-Assisted Human Evaluation of Machine Translation](https://mlsanity.com/paper/2406.12419/) (2024)
- [Pitfalls and Outlooks in Using COMET](https://mlsanity.com/paper/2408.15366/) (2024)
- [Hearing to Translate: The Effectiveness of Speech Modality Integration into LLMs](https://mlsanity.com/paper/2512.16378/) (2025)
- [Beyond Human-Only: Evaluating Human-Machine Collaboration for Collecting High-Quality Translation Data](https://mlsanity.com/paper/2410.11056/) (2024)
- [Contextual Metric Meta-Evaluation by Measuring Local Metric Accuracy](https://mlsanity.com/paper/2503.19828/) (2025)
- [COMET-poly: Machine Translation Metric Grounded in Other Candidates](https://mlsanity.com/paper/2508.18549/) (2025)

## Abstract (arXiv)

Automatic machine translation metrics typically rely on human translations to determine the quality of system translations. Common wisdom in the field dictates that the human references should be of very high quality. However, there are no cost-benefit analyses that could be used to guide practitioners who plan to collect references for machine translation evaluation. We find that higher-quality references lead to better metric correlations with humans at the segment-level. Having up to 7 references per segment and taking their average (or maximum) helps all metrics. Interestingly, the references from vendors of different qualities can be mixed together and improve metric success. Higher quality references, however, cost more to create and we frame this as an optimization problem: given a specific budget, what references should be collected to maximize metric success. These findings can be used by evaluators of shared tasks when references need to be created under a certain budget.