# Finding Replicable Human Evaluations via Stable Ranking Probability

arXiv: https://arxiv.org/abs/2404.01474 · HTML: https://mlsanity.com/paper/2404.01474/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 26 references that are in the map.
- First posted on arXiv 1 Apr 2024.

## Builds on (most cited first, 6 of 6)

- [Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation](https://mlsanity.com/paper/2104.14478/) (2021)
- [Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text](https://mlsanity.com/paper/2202.06935/) (2022)
- [Measuring Uncertainty in Translation Quality Evaluation (TQE)](https://mlsanity.com/paper/2111.07699/) (2021)
- [Consistent Human Evaluation of Machine Translation across Language Pairs](https://mlsanity.com/paper/2205.08533/) (2022)
- [Toward More Effective Human Evaluation for Machine Translation](https://mlsanity.com/paper/2204.05307/) (2022)
- [Searching for a higher power in the human evaluation of MT](https://mlsanity.com/paper/2210.11612/) (2022)

## Built on by (most cited first, 13 of 13)

- [Error Span Annotation: A Balanced Approach for Human Evaluation of Machine Translation](https://mlsanity.com/paper/2406.11580/) (2024)
- [Guardians of the Machine Translation Meta-Evaluation: Sentinel Metrics Fall In!](https://mlsanity.com/paper/2408.13831/) (2024)
- [AI-Assisted Human Evaluation of Machine Translation](https://mlsanity.com/paper/2406.12419/) (2024)
- [Has Machine Translation Evaluation Achieved Human Parity? The Human Reference and the Limits of Progress](https://mlsanity.com/paper/2506.19571/) (2025)
- [Enhancing Human Evaluation in Machine Translation with Comparative Judgment](https://mlsanity.com/paper/2502.17797/) (2025)
- [TranslateGemma Technical Report](https://mlsanity.com/paper/2601.09012/) (2026)
- [Estimating Machine Translation Difficulty](https://mlsanity.com/paper/2508.10175/) (2025)
- [From Jack of All Trades to Master of One: Specializing LLM-based Autoraters to a Test Set](https://mlsanity.com/paper/2411.15387/) (2024)
- [MQM Re-Annotation: A Technique for Collaborative Evaluation of Machine Translation](https://mlsanity.com/paper/2510.24664/) (2025)
- [Span-Level Machine Translation Meta-Evaluation](https://mlsanity.com/paper/2603.19921/) (2026)
- [Automatic Image-Level Morphological Trait Annotation for Organismal Images](https://mlsanity.com/paper/2604.01619/) (2026)
- [A Graph-Enhanced Defense Framework for Explainable Fake News Detection with LLM](https://mlsanity.com/paper/2604.06666/) (2026)
- [Same Winners, Different Success Rates: Evaluating How LLM Agents Recover from Failures](https://mlsanity.com/paper/2609.34215/) (2026)

## Abstract (arXiv)

Reliable human evaluation is critical to the development of successful natural language generation models, but achieving it is notoriously difficult. Stability is a crucial requirement when ranking systems by quality: consistent ranking of systems across repeated evaluations is not just desirable, but essential. Without it, there is no reliable foundation for hill-climbing or product launch decisions. In this paper, we use machine translation and its state-of-the-art human evaluation framework, MQM, as a case study to understand how to set up reliable human evaluations that yield stable conclusions. We investigate the optimal configurations for item allocation to raters, number of ratings per item, and score normalization. Our study on two language pairs provides concrete recommendations for designing replicable human evaluation studies. We also collect and release the largest publicly available dataset of multi-segment translations rated by multiple professional translators, consisting of nearly 140,000 segment annotations across two language pairs.