With a Little Help from the Authors: Reproducing Human Evaluation of an MT Error Detector
Not on the map. Why?
Abstract (arXiv)
This work presents our efforts to reproduce the results of the human evaluation experiment presented in the paper of Vamvas and Sennrich (2022), which evaluated an automatic system detecting over- and undertranslations (translations containing more or less information than the original) in machine translation (MT) outputs. Despite the high quality of the documentation and code provided by the authors, we discuss some problems we found in reproducing the exact experimental setup and offer recommendations for improving reproducibility. Our replicated results generally confirm the conclusions of the original study, but in some cases, statistically significant differences were observed, suggesting a high variability of human annotation.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 5 of its 17 references that are in the map. First posted on arXiv 12 Aug 2023.
Builds on
5 of 17 references.
References · 5
- 2020Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)cited by 322
- 2023Missing Information, Unresponsive Authors, Experimental Flaws: The Impossibility of Assessing the Reproducibility of Previous Human Evaluations in NLPcited by 21
- 2021The Human Evaluation Datasheet 1.0: A Template for Recording Details of Human Evaluation Experiments in NLPcited by 19
- 2022As Little as Possible, as Much as Necessary: Detecting Over- and Undertranslations with Contrastive Conditioningcited by 11
- 2022Quantified Reproducibility Assessment of NLP Resultscited by 8
Built on by
Not cited yet.