mlsanityData to 14 Oct 2026

Calibrated Preference Learning: The Case of Label Ranking

arXiv 2605.30447 ↗First posted 28 May 2026cs.LG cs.AI stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

Calibration, the alignment of predicted probabilities with true outcome frequencies, is essential for reliable decision-making. While extensively studied for classification and regression, calibration has not been formally addressed for probabilistic label ranking, where the goal is to predict a distribution over orderings of a label set. Naively treating rankings as classes ignores their structure and fails to capture important modalities such as pairwise and top-k predictions. We formalize calibration for label ranking and develop a hierarchy of notions covering full rankings, sub-rankings, and top-k rankings. We prove that full-rank calibration implies the others but not conversely, and sub-ranking and top-k calibration are incomparable. Empirically, we find popular label ranking models are often poorly calibrated, with substantial differences between sub-ranking and top-k metrics. Applying our framework to RLHF reward models, we find that calibration correlates strongly but not perfectly with benchmark accuracy, suggesting it captures a meaningful quality dimension beyond top-1 accuracy. These findings motivate future work on understanding the downstream effects of miscalibration and developing methods to correct it.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 34 of its 73 references that are in the map. First posted on arXiv 28 May 2026.

Builds on

34 of 73 references.

References · 34

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017On Calibration of Modern Neural Networkscited by 4,960
  4. 2019Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methodscited by 913
  5. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
  6. 2019Measuring Calibration in Deep Learningcited by 446
  7. 2020Calibrating Deep Neural Networks using Focal Losscited by 357
  8. 2019Evaluating model calibration in classificationcited by 168
  9. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 150
  10. 2019Distribution Calibration for Regressioncited by 89
  11. 2019Calibration tests in multi-class classification: A unifying frameworkcited by 79
  12. 2023Calibration in Deep Learning: A Survey of the State-of-the-Artcited by 75
22 more
  1. 2021Classifier Calibration: A survey on how to assess and improve predicted class probabilitiescited by 73
  2. 2023On the Calibration of Large Language Models and Alignmentcited by 73
  3. 2024Quantile Regression for Distributional Reward Models in RLHFcited by 60
  4. 2022A Consistent and Differentiable Lp Canonical Calibration Error Estimatorcited by 47
  5. 2022Efficient computation of rankings from pairwise comparisonscited by 31
  6. 2021On the Calibration and Uncertainty of Neural Learning to Rank Modelscited by 24
  7. 2022Calibration tests beyond classificationcited by 20
  8. 2018A Structured Prediction Approach for Label Rankingcited by 18
  9. 2021Obtaining Calibrated Probabilities with Personalized Ranking Modelscited by 15
  10. 2025Pairwise Calibrated Rewards for Pluralistic Alignmentcited by 14
  11. 2022On the Calibration of Probabilistic Classifier Setscited by 8
  12. 2025A calibration test for evaluating set-based epistemic uncertainty representationscited by 7
  13. 2023From Classification Accuracy to Proper Scoring Rules: Elicitability of Probabilistic Top List Predictionscited by 5
  14. 2025Random Forest Calibrationcited by 3
  15. 2025Calibrated Recommendations: Survey and Future Directionscited by 3
  16. 2022Heuristic Search for Rank Aggregation with Application to Label Rankingcited by 2
  17. 2024A Self-boosted Framework for Calibrated Rankingcited by 2
  18. 2024Calibrating the Predictions for Top-N Recommendationscited by 2
  19. 2025A comprehensive review of classifier probability calibration metricscited by 2
  20. 2017Statistical Inference for Incomplete Ranking Data: The Case of Rank-Dependent Coarseningcited by 1
  21. 2020Supporting Hard Queries over Probabilistic Preferencescited by 1
  22. 2025ResponseRank: Data-Efficient Reward Modeling through Preference Strength Learningcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Calibrate, Don't Curate: Label-Efficient Estimation from Noisy LLM Judges
  2. 2025Similarity as Reward Alignment: Robust and Versatile Preference-based Reinforcement Learning
  3. 2025DCRM: A Heuristic to Measure Response Pair Quality in Preference Optimization
  4. 2025Probing Preference Representations: A Multi-Dimensional Evaluation and Analysis Method for Reward Models
  5. 2024Preference Learning Algorithms Do Not Learn Preference Rankings
  6. 2026Reward Learning from Best-of-N Preference Data: Targets, Tradeoffs, and Design Principles
  7. 2025Judging with Confidence: Calibrating Autoraters to Preference Distributions
  8. 2026Prompt-Dependent Ranking of Large Language Models with Uncertainty Quantification