# PEAR: Pairwise Evaluation for Automatic Relative Scoring in Machine Translation

arXiv: https://arxiv.org/abs/2601.18006 · HTML: https://mlsanity.com/paper/2601.18006/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 61 references that are in the map.
- First posted on arXiv 25 Jan 2026.

## Builds on (most cited first, 30 of 37)

- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Large Language Models Are State-of-the-Art Evaluators of Translation Quality](https://mlsanity.com/paper/2302.14520/) (2023)
- [Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation](https://mlsanity.com/paper/2104.14478/) (2021)
- [xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detection](https://mlsanity.com/paper/2310.10482/) (2023)
- [InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training](https://mlsanity.com/paper/2007.07834/) (2020)
- [CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task](https://mlsanity.com/paper/2209.06243/) (2022)
- [To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translation](https://mlsanity.com/paper/2107.10821/) (2021)
- [GEMBA-MQM: Detecting Translation Quality Error Spans with GPT-4](https://mlsanity.com/paper/2310.13988/) (2023)
- [High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metrics](https://mlsanity.com/paper/2111.09388/) (2021)
- [The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation](https://mlsanity.com/paper/2109.06835/) (2021)
- [MetricX-24: The Google Submission to the WMT 2024 Metrics Shared Task](https://mlsanity.com/paper/2410.03983/) (2024)
- [Scaling up COMETKIWI: Unbabel-IST 2023 Submission for the Quality Estimation Shared Task](https://mlsanity.com/paper/2309.11925/) (2023)
- [Larger-Scale Transformers for Multilingual Masked Language Modeling](https://mlsanity.com/paper/2105.00572/) (2021)
- [Ties Matter: Meta-Evaluating Modern Metrics with Pairwise Accuracy and Tie Calibration](https://mlsanity.com/paper/2305.14324/) (2023)
- [A Statistical Analysis of Summarization Evaluation Metrics using Resampling Methods](https://mlsanity.com/paper/2104.00054/) (2021)
- [Scientific Credibility of Machine Translation Research: A Meta-Evaluation of 769 Papers](https://mlsanity.com/paper/2106.15195/) (2021)
- [Error Span Annotation: A Balanced Approach for Human Evaluation of Machine Translation](https://mlsanity.com/paper/2406.11580/) (2024)
- [Navigating the Metrics Maze: Reconciling Score Magnitudes and Accuracies](https://mlsanity.com/paper/2401.06760/) (2024)
- [Efficient Online Scalar Annotation with Bounded Support](https://mlsanity.com/paper/1806.01170/) (2018)
- [IndicMT Eval: A Dataset to Meta-Evaluate Machine Translation metrics for Indian Languages](https://mlsanity.com/paper/2212.10180/) (2022)
- [Learning to Evaluate Translation Beyond English: BLEURT Submissions to the WMT Metrics 2020 Shared Task](https://mlsanity.com/paper/2010.04297/) (2020)
- [Improving Statistical Significance in Human Evaluation of Automatic Metrics via Soft Pairwise Accuracy](https://mlsanity.com/paper/2409.09598/) (2024)
- [How to Select Datapoints for Efficient Human Evaluation of NLG Models?](https://mlsanity.com/paper/2501.18251/) (2025)
- [Guardians of the Machine Translation Meta-Evaluation: Sentinel Metrics Fall In!](https://mlsanity.com/paper/2408.13831/) (2024)
- [AI-Assisted Human Evaluation of Machine Translation](https://mlsanity.com/paper/2406.12419/) (2024)
- [MT-Ranker: Reference-free machine translation evaluation by inter-system ranking](https://mlsanity.com/paper/2401.17099/) (2024)

## Built on by (most cited first, 1 of 1)

- [LocQE: Principled Domain Adaptation for Localisation Quality Estimation by Leveraging Post-Edits](https://mlsanity.com/paper/2609.18720/) (2026)

## Abstract (arXiv)

We present PEAR (Pairwise Evaluation for Automatic Relative Scoring), a supervised quality estimation (QE) metric family that reframes reference-free machine translation (MT) evaluation as a graded pairwise comparison. Given a source segment and two candidate translations, PEAR predicts the direction and magnitude of their quality difference. The metrics are trained using pairwise supervision derived from differences in human judgments, with an additional regularization term that encourages sign inversion under candidate order reversal. On the WMT24 meta-evaluation benchmark, PEAR outperforms strictly matched single-candidate QE baselines trained with the same data and backbones, isolating the benefit of the proposed pairwise formulation. Despite using substantially fewer parameters than recent large metrics, PEAR surpasses far larger QE models and reference-based metrics. Our analysis further indicates that PEAR yields a less redundant evaluation signal relative to other top metrics. Finally, we show that PEAR is an effective utility function for minimum Bayes risk (MBR) decoding, reducing pairwise scoring cost at negligible impact.