# Don't Sweat the Small Stuff: Segment-Level Meta-Evaluation Based on Pairwise Difference Correlation

arXiv: https://arxiv.org/abs/2509.25546 · HTML: https://mlsanity.com/paper/2509.25546/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 15 references that are in the map.
- First posted on arXiv 29 Sep 2025.

## Builds on (most cited first, 5 of 5)

- [Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation](https://mlsanity.com/paper/2104.14478/) (2021)
- [Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics](https://mlsanity.com/paper/2006.06264/) (2020)
- [Ties Matter: Meta-Evaluating Modern Metrics with Pairwise Accuracy and Tie Calibration](https://mlsanity.com/paper/2305.14324/) (2023)
- [Guardians of the Machine Translation Meta-Evaluation: Sentinel Metrics Fall In!](https://mlsanity.com/paper/2408.13831/) (2024)
- [Has Machine Translation Evaluation Achieved Human Parity? The Human Reference and the Limits of Progress](https://mlsanity.com/paper/2506.19571/) (2025)

## Built on by (most cited first, 2 of 2)

- [Which Metrics Save the Most Human Annotation? Prediction-Powered Evaluation and Meta-Evaluation](https://mlsanity.com/paper/2608.26638/) (2026)
- [LocQE: Principled Domain Adaptation for Localisation Quality Estimation by Leveraging Post-Edits](https://mlsanity.com/paper/2609.18720/) (2026)

## Abstract (arXiv)

This paper introduces Pairwise Difference Pearson (PDP), a novel segment-level meta-evaluation metric for Machine Translation (MT) that address limitations in previous Pearson's ρ-based and and Kendall's τ-based meta-evaluation approaches. PDP is a correlation-based metric that utilizes pairwise differences rather than raw scores. It draws on information from all segments for a more robust understanding of score distributions and uses segment-wise pairwise differences to refine Global Pearson to intra-segment score comparisons. Analysis on the WMT'24 shared task shows PDP properly ranks sentinel evaluation metrics and better aligns with human error weightings than previous work. Noise injection analysis demonstrates PDP's robustness to random noise, segment bias, and system bias while highlighting its sensitivity to extreme outliers.