mlsanityData to 7 Oct 2026

Improving Statistical Significance in Human Evaluation of Automatic Metrics via Soft Pairwise Accuracy

arXiv 2409.09598 ↗First posted 15 Sep 2024cs.CL cs.AIcited by 25

Not on the map. Why?

Abstract (arXiv)

Selecting an automatic metric that best emulates human annotators is often non-trivial, because there is no clear definition of "best emulates." A meta-metric is required to compare the human judgments to the automatic metric scores, and metric rankings depend on the choice of meta-metric. We propose Soft Pairwise Accuracy (SPA), a new meta-metric that builds on Pairwise Accuracy (PA) but incorporates the statistical significance of both the human judgments and the metric scores. We show that SPA is more stable than PA with respect to changes in the number of systems/segments used for evaluation. We also show that PA can only assign a small set of distinct output values to metrics, and this results in many metrics being artificially assigned the exact same PA score. We demonstrate that SPA fixes this issue. Finally, we show that SPA is more discriminative than PA, producing more statistically significant comparisons between metrics. SPA was selected as the official system-level metric for the 2024 WMT Metrics Shared Task.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 13 of its 49 references that are in the map. First posted on arXiv 15 Sep 2024.

Builds on

13 of 49 references.

References · 13

  1. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
  2. 2023xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detectioncited by 260
  3. 2021To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translationcited by 161
  4. 2020Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metricscited by 155
  5. 2020Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasingcited by 114
  6. 2021A Statistical Analysis of Summarization Evaluation Metrics using Resampling Methodscited by 71
  7. 2023Ties Matter: Meta-Evaluating Modern Metrics with Pairwise Accuracy and Tie Calibrationcited by 71
  8. 2022Embarrassingly Easy Document-Level MT Metrics: How to Convert Any Pretrained Metric Into a Document-Level Metriccited by 56
  9. 2020Paraphrase Generation as Zero-Shot Multilingual Translation: Disentangling Semantic Similarity from Lexical and Syntactic Diversitycited by 35
  10. 2024Fine-Tuned Machine Translation Metrics Struggle in Unseen Domainscited by 21
  11. 2023SLIDE: Reference-free Evaluation for Machine Translation using a Sliding Document Windowcited by 11
  12. 2023Training and Meta-Evaluating Machine Translation Evaluation Metrics at the Paragraph Levelcited by 9
1 more
  1. 2024Is Context Helpful for Chat Translation Evaluation?cited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 13; 2026: 9.

25 papers.

Sort:

By area:

  1. 2025How to Select Datapoints for Efficient Human Evaluation of NLG Models?cited by 23
  2. 2025Zero-shot Benchmarking: A Framework for Flexible and Scalable Automatic Evaluation of Language Modelscited by 14
  3. 2024Watching the Watchers: Exposing Gender Disparities in Machine Translation Quality Estimationcited by 10
  4. 2025Has Machine Translation Evaluation Achieved Human Parity? The Human Reference and the Limits of Progresscited by 10
  5. 2025MetricX-25 and GemSpanEval: Google Translate Submissions to the WMT25 Evaluation Shared Taskcited by 10
  6. 2024Adapters for Altering LLM Vocabularies: What Languages Benefit the Most?cited by 9
  7. 2025Remedy: Learning Machine Translation Evaluation from Human Preferences with Reward Modelingcited by 9
  8. 2025MindEval: Benchmarking Language Models on Multi-turn Mental Health Supportcited by 9
  9. 2024MetaMetrics-MT: Tuning Meta-Metrics for Machine Translation via Human Preference Calibrationcited by 6
  10. 2025Adding Chocolate to Mint: Mitigating Metric Interference in Machine Translationcited by 6
  11. 2026Beyond Literal Mapping: Benchmarking and Improving Non-Literal Translation Evaluationcited by 6
  12. 2025Feeding Two Birds or Favoring One? Adequacy-Fluency Tradeoffs in Evaluation and Meta-Evaluation of Machine Translationcited by 3
13 more
  1. 2025Reward Models are Metrics in a Trench Coatcited by 2
  2. 2026Who Watches the Watchmen? Humans Disagree With Translation Metrics on Unseen Domainscited by 2
  3. 2025Same evaluation, more tokens: On the effect of input length for machine translation evaluation using Large Language Modelscited by 1
  4. 2025Remedy-R: Generative Reasoning for Machine Translation Evaluation without Error Annotationscited by 1
  5. 2026PEAR: Pairwise Evaluation for Automatic Relative Scoring in Machine Translationcited by 1
  6. 2026Why We Need Speech to Evaluate Speech Translationcited by 1
  7. 2026Which Metrics Save the Most Human Annotation? Prediction-Powered Evaluation and Meta-Evaluationcited by 1
  8. 2025Evaluating Language Translation Models by Playing Telephonecited by 0
  9. 2025Minimum Bayes Risk Decoding for Error Span Detection in Reference-Free Automatic Machine Translation Evaluationcited by 0
  10. 2026Is Human Annotation Necessary? Iterative MBR Distillation for Error Span Detection in Machine Translationcited by 0
  11. 2026Dynamic Meta-Metrics: Source-Sentence Conditioned Weighting for MT Evaluationcited by 0
  12. 2026CompactQE: Interpretable Translation Quality Estimation via Small Open-Weight LLMscited by 0
  13. 2026Mind Which Bird You Favour: Parameterizing Adequacy-Fluency Balance in Meta-Evaluation of Machine Translationcited by 0