# Calibrated Preference Learning: The Case of Label Ranking

arXiv: https://arxiv.org/abs/2605.30447 · HTML: https://mlsanity.com/paper/2605.30447/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 34 of its 73 references that are in the map.
- First posted on arXiv 28 May 2026.

## Builds on (most cited first, 30 of 34)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods](https://mlsanity.com/paper/1910.09457/) (2019)
- [Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback](https://mlsanity.com/paper/2305.14975/) (2023)
- [Measuring Calibration in Deep Learning](https://mlsanity.com/paper/1904.01685/) (2019)
- [Calibrating Deep Neural Networks using Focal Loss](https://mlsanity.com/paper/2002.09437/) (2020)
- [Evaluating model calibration in classification](https://mlsanity.com/paper/1902.06977/) (2019)
- [RewardBench 2: Advancing Reward Model Evaluation](https://mlsanity.com/paper/2506.01937/) (2025)
- [Distribution Calibration for Regression](https://mlsanity.com/paper/1905.06023/) (2019)
- [Calibration tests in multi-class classification: A unifying framework](https://mlsanity.com/paper/1910.11385/) (2019)
- [Calibration in Deep Learning: A Survey of the State-of-the-Art](https://mlsanity.com/paper/2308.01222/) (2023)
- [Classifier Calibration: A survey on how to assess and improve predicted class probabilities](https://mlsanity.com/paper/2112.10327/) (2021)
- [On the Calibration of Large Language Models and Alignment](https://mlsanity.com/paper/2311.13240/) (2023)
- [Quantile Regression for Distributional Reward Models in RLHF](https://mlsanity.com/paper/2409.10164/) (2024)
- [A Consistent and Differentiable Lp Canonical Calibration Error Estimator](https://mlsanity.com/paper/2210.07810/) (2022)
- [Efficient computation of rankings from pairwise comparisons](https://mlsanity.com/paper/2207.00076/) (2022)
- [On the Calibration and Uncertainty of Neural Learning to Rank Models](https://mlsanity.com/paper/2101.04356/) (2021)
- [Calibration tests beyond classification](https://mlsanity.com/paper/2210.13355/) (2022)
- [A Structured Prediction Approach for Label Ranking](https://mlsanity.com/paper/1807.02374/) (2018)
- [Obtaining Calibrated Probabilities with Personalized Ranking Models](https://mlsanity.com/paper/2112.07428/) (2021)
- [Pairwise Calibrated Rewards for Pluralistic Alignment](https://mlsanity.com/paper/2506.06298/) (2025)
- [On the Calibration of Probabilistic Classifier Sets](https://mlsanity.com/paper/2205.10082/) (2022)
- [A calibration test for evaluating set-based epistemic uncertainty representations](https://mlsanity.com/paper/2502.16299/) (2025)
- [From Classification Accuracy to Proper Scoring Rules: Elicitability of Probabilistic Top List Predictions](https://mlsanity.com/paper/2301.11797/) (2023)
- [Random Forest Calibration](https://mlsanity.com/paper/2501.16756/) (2025)
- [Calibrated Recommendations: Survey and Future Directions](https://mlsanity.com/paper/2507.02643/) (2025)
- [Heuristic Search for Rank Aggregation with Application to Label Ranking](https://mlsanity.com/paper/2201.03893/) (2022)
- [A Self-boosted Framework for Calibrated Ranking](https://mlsanity.com/paper/2406.08010/) (2024)
- [Calibrating the Predictions for Top-N Recommendations](https://mlsanity.com/paper/2408.11596/) (2024)

## Abstract (arXiv)

Calibration, the alignment of predicted probabilities with true outcome frequencies, is essential for reliable decision-making. While extensively studied for classification and regression, calibration has not been formally addressed for probabilistic label ranking, where the goal is to predict a distribution over orderings of a label set. Naively treating rankings as classes ignores their structure and fails to capture important modalities such as pairwise and top-k predictions. We formalize calibration for label ranking and develop a hierarchy of notions covering full rankings, sub-rankings, and top-k rankings. We prove that full-rank calibration implies the others but not conversely, and sub-ranking and top-k calibration are incomparable. Empirically, we find popular label ranking models are often poorly calibrated, with substantial differences between sub-ranking and top-k metrics. Applying our framework to RLHF reward models, we find that calibration correlates strongly but not perfectly with benchmark accuracy, suggesting it captures a meaningful quality dimension beyond top-1 accuracy. These findings motivate future work on understanding the downstream effects of miscalibration and developing methods to correct it.