# Pair Difficulty Matters: Rethinking Pairwise LLM-as-a-Judge Evaluation and Consistency

arXiv: https://arxiv.org/abs/2609.37577 · HTML: https://mlsanity.com/paper/2609.37577/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 3 of its 19 references that are in the map.
- First posted on arXiv 29 Sep 2026.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 3 of 3)

- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [Ministral 3](https://mlsanity.com/paper/2601.08584/) (2026)
- [Are We on the Right Way to Assessing LLM-as-a-Judge?](https://mlsanity.com/paper/2512.16041/) (2025)

## Built on by (most cited first, 1 of 1)

- [Cheap, Consistent, Calibrated, but Biased? Typed Decision Models as Judges](https://mlsanity.com/paper/2610.18122/) (2026)

## Abstract (arXiv)

Large Language Model judges are widely used to rank texts and text-generating systems through pairwise comparison, and their reliability is typically assessed via three proxies: position bias, transitivity, and pairwise agreement (self- or human-labeled). Because these proxies drive judge selection and benchmarking, a substantial literature reporting that judges perform poorly on them risks steering practitioners away from otherwise capable evaluators. We argue this assessment is misleading. Under the Bradley--Terry geometry underlying pairwise aggregation, each proxy is dominated by close-rank-gap pairs, where inconsistency is information-theoretically expected and individual verdicts contribute little to the aggregate ranking; far-gap pairs carry the ranking signal but barely move the proxies. We formalize this argument and validate it in a controlled simulation and on two human-rated corpora: the proxies correlate only weakly with ranking accuracy against gold, and their predictive component concentrates in the far-gap regime. Judges should therefore be assessed on rank-gap-conditional metrics, ideally against human rankings. Code at https://github.com/brunobrocai/PairDifficulty.