mlsanityData to 14 Oct 2026

Ties Matter: Meta-Evaluating Modern Metrics with Pairwise Accuracy and Tie Calibration

arXiv 2305.14324 ↗First posted 23 May 2023cs.CLcited by 74

Not on the map. Why?

Abstract (arXiv)

Kendall's tau is frequently used to meta-evaluate how well machine translation (MT) evaluation metrics score individual translations. Its focus on pairwise score comparisons is intuitive but raises the question of how ties should be handled, a gray area that has motivated different variants in the literature. We demonstrate that, in settings like modern MT meta-evaluation, existing variants have weaknesses arising from their handling of ties, and in some situations can even be gamed. We propose instead to meta-evaluate metrics with a version of pairwise accuracy that gives metrics credit for correctly predicting ties, in combination with a tie calibration procedure that automatically introduces ties into metric scores, enabling fair comparison between metrics that do and do not predict ties. We argue and provide experimental evidence that these modifications lead to fairer ranking-based assessments of metric performance.

Key facts. Cited by 74 papers in the map (data through 14 Oct 2026). Builds on 9 of its 23 references that are in the map. First posted on arXiv 23 May 2023.

Builds on

9 of 23 references.

References · 9

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  3. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  4. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 420
  5. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 338
  6. 2021To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translationcited by 163
  7. 2020Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metricscited by 158
  8. 2021High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metricscited by 104
  9. 2022Quality-Aware Decoding for Neural Machine Translationcited by 49

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 22; 2025: 27; 2026: 22.

74 papers.

Sort:

By area:

  1. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 474
  2. 2025Improving Video Generation with Human Feedbackcited by 284
  3. 2023xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detectioncited by 265
  4. 2024VisionReward: Fine-Grained Multi-Dimensional Human Preference Learning for Image and Video Generationcited by 156
  5. 2024GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generationcited by 135
  6. 2024MetricX-24: The Google Submission to the WMT 2024 Metrics Shared Taskcited by 93
  7. 2025R2I-Bench: Benchmarking Reasoning-Driven Text-to-Image Generationcited by 32
  8. 2024Beyond correlation: The Impact of Human Uncertainty in Measuring the Effectiveness of Automatic Evaluation and LLM-as-a-Judgecited by 31
  9. 2025VQ-Insight: Teaching VLMs for AI-Generated Video Quality Understanding via Progressive Visual Reinforcement Learningcited by 26
  10. 2024Improving Statistical Significance in Human Evaluation of Automatic Metrics via Soft Pairwise Accuracycited by 25
  11. 2023AfriMTE and AfriCOMET: Enhancing COMET to Embrace Under-resourced African Languagescited by 24
  12. 2025How to Select Datapoints for Efficient Human Evaluation of NLG Models?cited by 23
38 more
  1. 2024MQM-APE: Toward High-Quality Error Annotation Predictors with Automatic Post-Editing in LLM Translation Evaluatorscited by 22
  2. 2025Cycle Consistency as Reward: Learning Image-Text Alignment without Human Preferencescited by 22
  3. 2024Guardians of the Machine Translation Meta-Evaluation: Sentinel Metrics Fall In!cited by 21
  4. 2024FactPICO: Factuality Evaluation for Plain Language Summarization of Medical Evidencecited by 18
  5. 2024Can Automatic Metrics Assess High-Quality Translations?cited by 16
  6. 2024PrExMe! Large Scale Prompt Exploration of Open Source LLMs for Machine Translation and Summarization Evaluationcited by 16
  7. 2024How Reliable Are Automatic Evaluation Methods for Instruction-Tuned LLMs?cited by 13
  8. 2024M-MAD: Multidimensional Multi-Agent Debate for Advanced Machine Translation Evaluationcited by 13
  9. 2025Omni-Reward: Towards Generalist Omni-Modal Reward Modeling with Free-Form Preferencescited by 11
  10. 2024Beyond Correlation: Interpretable Evaluation of Machine Translation Metricscited by 10
  11. 2024Watching the Watchers: Exposing Gender Disparities in Machine Translation Quality Estimationcited by 10
  12. 2025LiTransProQA: an LLM-based Literary Translation evaluation metric with Professional Question Answeringcited by 10
  13. 2025Has Machine Translation Evaluation Achieved Human Parity? The Human Reference and the Limits of Progresscited by 10
  14. 2025Multimodal LLMs as Customized Reward Models for Text-to-Image Generationcited by 10
  15. 2025MetricX-25 and GemSpanEval: Google Translate Submissions to the WMT25 Evaluation Shared Taskcited by 10
  16. 2025MindEval: Benchmarking Language Models on Multi-turn Mental Health Supportcited by 10
  17. 2023Training and Meta-Evaluating Machine Translation Evaluation Metrics at the Paragraph Levelcited by 9
  18. 2025Remedy: Learning Machine Translation Evaluation from Human Preferences with Reward Modelingcited by 9
  19. 2024xCOMET-lite: Bridging the Gap Between Efficiency and Quality in Learned MT Evaluation Metricscited by 7
  20. 2025Enhancing Human Evaluation in Machine Translation with Comparative Judgmentcited by 7
  21. 2024MetaMetrics-MT: Tuning Meta-Metrics for Machine Translation via Human Preference Calibrationcited by 6
  22. 2025Adding Chocolate to Mint: Mitigating Metric Interference in Machine Translationcited by 6
  23. 2025Flex-Judge: Text-Only Reasoning Unleashes Zero-Shot Multimodal Evaluatorscited by 6
  24. 2025Identifying Reliable Evaluation Metrics for Scientific Text Revisioncited by 6
  25. 2025Estimating Machine Translation Difficultycited by 6
  26. 2026Beyond Literal Mapping: Benchmarking and Improving Non-Literal Translation Evaluationcited by 6
  27. 2024Is Context Helpful for Chat Translation Evaluation?cited by 5
  28. 2026ChartE3: A Comprehensive Benchmark for End-to-End Chart Editingcited by 5
  29. 2024Analyzing and Evaluating Correlation Measures in NLG Meta-Evaluationcited by 4
  30. 2024PromptOptMe: Error-Aware Prompt Compression for LLM-based MT Evaluation Metricscited by 4
  31. 2025Feeding Two Birds or Favoring One? Adequacy-Fluency Tradeoffs in Evaluation and Meta-Evaluation of Machine Translationcited by 4
  32. 2026Building a Precise Video Language with Human-AI Oversightcited by 3
  33. 2024Uncovering Factor Level Preferences to Improve Human-Model Alignmentcited by 2
  34. 2024From Jack of All Trades to Master of One: Specializing LLM-based Autoraters to a Test Setcited by 2
  35. 2025CROC: Evaluating and Training T2I Metrics with Pseudo- and Human-Labeled Contrastive Robustness Checkscited by 2
  36. 2025Don't Sweat the Small Stuff: Segment-Level Meta-Evaluation Based on Pairwise Difference Correlationcited by 2
  37. 2025MQM Re-Annotation: A Technique for Collaborative Evaluation of Machine Translationcited by 2
  38. 2026Prototypicality Bias Reveals Blindspots in Multimodal Evaluation Metricscited by 2

50 most cited of 74 All 74 (JSON)