# Toward More Effective Human Evaluation for Machine Translation

arXiv: https://arxiv.org/abs/2204.05307 · HTML: https://mlsanity.com/paper/2204.05307/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 24 references that are in the map.
- First posted on arXiv 11 Apr 2022.

## Builds on (most cited first, 13 of 13)

- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation](https://mlsanity.com/paper/2104.14478/) (2021)
- [Evaluation of Text Generation: A Survey](https://mlsanity.com/paper/2006.14799/) (2020)
- [Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text](https://mlsanity.com/paper/2202.06935/) (2022)
- [Unifying Human and Statistical Evaluation for Natural Language Generation](https://mlsanity.com/paper/1904.02792/) (2019)
- [The price of debiasing automatic metrics in natural language evaluation](https://mlsanity.com/paper/1807.02202/) (2018)
- [The statistical advantage of automatic NLG metrics at the system level](https://mlsanity.com/paper/2105.12437/) (2021)
- [Measuring Uncertainty in Translation Quality Evaluation (TQE)](https://mlsanity.com/paper/2111.07699/) (2021)
- [Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort](https://mlsanity.com/paper/2105.13385/) (2021)
- [Using Sampling to Estimate and Improve Performance of Automated Scoring Systems with Guarantees](https://mlsanity.com/paper/2111.08906/) (2021)
- [Improving Human-Labeled Data through Dynamic Automatic Conflict Resolution](https://mlsanity.com/paper/2012.04169/) (2020)
- [Dynamic Human Evaluation for Relative Model Comparisons](https://mlsanity.com/paper/2112.08048/) (2021)

## Built on by (most cited first, 10 of 10)

- [How to Select Datapoints for Efficient Human Evaluation of NLG Models?](https://mlsanity.com/paper/2501.18251/) (2025)
- [Finding Replicable Human Evaluations via Stable Ranking Probability](https://mlsanity.com/paper/2404.01474/) (2024)
- [Angler: Helping Machine Translation Practitioners Prioritize Model Improvements](https://mlsanity.com/paper/2304.05967/) (2023)
- [Pearmut: Human Evaluation of Translation Made Trivial](https://mlsanity.com/paper/2601.02933/) (2026)
- [Translation in the Hands of Many:Centering Lay Users in Machine Translation Interactions](https://mlsanity.com/paper/2502.13780/) (2025)
- [Searching for a higher power in the human evaluation of MT](https://mlsanity.com/paper/2210.11612/) (2022)
- [Measuring Fine-Grained Semantic Equivalence with Abstract Meaning Representation](https://mlsanity.com/paper/2210.03018/) (2022)
- [MQM Re-Annotation: A Technique for Collaborative Evaluation of Machine Translation](https://mlsanity.com/paper/2510.24664/) (2025)
- [Dynamically Allocating Evaluation Effort for Model Ranking](https://mlsanity.com/paper/2608.03437/) (2026)
- [Beyond Scalar Scores: Exploring LLM-based Metrics for Clinical Significance Evaluation in Radiology Reports](https://mlsanity.com/paper/2606.18797/) (2026)

## Abstract (arXiv)

Improvements in text generation technologies such as machine translation have necessitated more costly and time-consuming human evaluation procedures to ensure an accurate signal. We investigate a simple way to reduce cost by reducing the number of text segments that must be annotated in order to accurately predict a score for a complete test set. Using a sampling approach, we demonstrate that information from document membership and automatic metrics can help improve estimates compared to a pure random sampling baseline. We achieve gains of up to 20% in average absolute error by leveraging stratified sampling and control variates. Our techniques can improve estimates made from a fixed annotation budget, are easy to implement, and can be applied to any problem with structure similar to the one we study.