A Measure of the System Dependence of Automated Metrics
Not on the map. Why?
Abstract (arXiv)
Automated metrics for Machine Translation have made significant progress, with the goal of replacing expensive and time-consuming human evaluations. These metrics are typically assessed by their correlation with human judgments, which captures the monotonic relationship between human and metric scores. However, we argue that it is equally important to ensure that metrics treat all systems fairly and consistently. In this paper, we introduce a method to evaluate this aspect.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 14 of its 26 references that are in the map. First posted on arXiv 4 Dec 2024.
Builds on
14 of 26 references.
References · 14
- 2017Attention Is All You Needcited by 60,361
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2023How Good Are GPT Models at Machine Translation? A Comprehensive Evaluationcited by 348
- 2023xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detectioncited by 265
- 2021To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translationcited by 163
- 2023GEMBA-MQM: Detecting Translation Quality Error Spans with GPT-4cited by 146
- 2020Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasingcited by 115
- 2018The price of debiasing automatic metrics in natural language evaluationcited by 60
- 2020Paraphrase Generation as Zero-Shot Multilingual Translation: Disentangling Semantic Similarity from Lexical and Syntactic Diversitycited by 35
- 2021Estimating Expected Calibration Errorscited by 24
- 2021The statistical advantage of automatic NLG metrics at the system levelcited by 18
- 2024Calibrate-Extrapolate: Rethinking Prevalence Estimation with Black Box Classifierscited by 6
Built on by
Citing papers per year: 2024: 1; 2025: 1.
2 papers.