# Heterogeneous Judge-Aware Ranking with Sensitivity, Disagreement, and Confidence

arXiv: https://arxiv.org/abs/2605.05073 · HTML: https://mlsanity.com/paper/2605.05073/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 May 2026.

## Built on by (most cited first, 3 of 3)

- [A Finite-Calibration Regime Map for LLM Judge Panels](https://mlsanity.com/paper/2606.01034/) (2026)
- [OSCAR: Order-aware Sensitivity Correction for AI Rankings](https://mlsanity.com/paper/2609.24128/) (2026)
- [DIAL: Position-Debiased LLM Judges with Adaptive Human Preference Calibration](https://mlsanity.com/paper/2609.31215/) (2026)

## Abstract (arXiv)

Pairwise comparisons from multiple judges are central to large language model evaluation and preference modeling, yet standard ranking pipelines often pool judgments into a single score vector, treating systematic judge disagreement as noise. We propose Heterogeneous Judge-Aware (HJA) ranking, a structured multi-judge ranking framework that separates consensus ranking, judge-specific sensitivity to consensus, and residual preference disagreement. HJA thereby treats ranking, judge sensitivity, and structured disagreement as separate inferential targets. We establish conditions under which this decomposition is identifiable and develop an anchored alternating algorithm that preserves the identifying geometry. For confidence quantification, we study a fixed-panel repeated-comparison regime in which the judge panel may remain fixed or modest while information grows through repeated judgments. This yields uncertainty statements for consensus and judge-specific ranking contrasts, sensitivity parameters, pairwise probabilities, and summaries of residual disagreement.Experiments on synthetic and real multi-judge comparison data show that HJA improves recovery, robustness, uncertainty calibration, and near-tie performance relative to pooled and sensitivity-only baselines. The fitted model also provides diagnostics for judge disagreement and model-affinity patterns, giving a statistically grounded framework for ranking under heterogeneous comparative judgments.