# MQM Re-Annotation: A Technique for Collaborative Evaluation of Machine Translation

arXiv: https://arxiv.org/abs/2510.24664 · HTML: https://mlsanity.com/paper/2510.24664/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 27 references that are in the map.
- First posted on arXiv 28 Oct 2025.

## Builds on (most cited first, 13 of 13)

- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation](https://mlsanity.com/paper/2104.14478/) (2021)
- [xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detection](https://mlsanity.com/paper/2310.10482/) (2023)
- [GEMBA-MQM: Detecting Translation Quality Error Spans with GPT-4](https://mlsanity.com/paper/2310.13988/) (2023)
- [The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluation](https://mlsanity.com/paper/2308.07286/) (2023)
- [MetricX-24: The Google Submission to the WMT 2024 Metrics Shared Task](https://mlsanity.com/paper/2410.03983/) (2024)
- [Ties Matter: Meta-Evaluating Modern Metrics with Pairwise Accuracy and Tie Calibration](https://mlsanity.com/paper/2305.14324/) (2023)
- [Error Span Annotation: A Balanced Approach for Human Evaluation of Machine Translation](https://mlsanity.com/paper/2406.11580/) (2024)
- [AI-Assisted Human Evaluation of Machine Translation](https://mlsanity.com/paper/2406.12419/) (2024)
- [Finding Replicable Human Evaluations via Stable Ranking Probability](https://mlsanity.com/paper/2404.01474/) (2024)
- [Toward More Effective Human Evaluation for Machine Translation](https://mlsanity.com/paper/2204.05307/) (2022)
- [Enhancing Human Evaluation in Machine Translation with Comparative Judgment](https://mlsanity.com/paper/2502.17797/) (2025)
- [Beyond Human-Only: Evaluating Human-Machine Collaboration for Collecting High-Quality Translation Data](https://mlsanity.com/paper/2410.11056/) (2024)

## Built on by (most cited first, 2 of 2)

- [Dynamically Allocating Evaluation Effort for Model Ranking](https://mlsanity.com/paper/2608.03437/) (2026)
- [Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged](https://mlsanity.com/paper/2610.10918/) (2026)

## Abstract (arXiv)

Human evaluation of machine translation is in an arms race with translation model quality: as our models get better, our evaluation methods need to be improved to ensure that quality gains are not lost in evaluation noise. To this end, we experiment with a two-stage version of the current state-of-the-art translation evaluation paradigm (MQM), which we call MQM re-annotation. In this setup, an MQM annotator reviews and edits a set of pre-existing MQM annotations, that may have come from themselves, another human annotator, or an automatic MQM annotation system. We demonstrate that rater behavior in re-annotation aligns with our goals, and that re-annotation results in higher-quality annotations, mostly due to finding errors that were missed during the first pass.