# Obtaining Calibrated Probabilities with Personalized Ranking Models

arXiv: https://arxiv.org/abs/2112.07428 · HTML: https://mlsanity.com/paper/2112.07428/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 29 references that are in the map.
- First posted on arXiv 9 Dec 2021.

## Builds on (most cited first, 8 of 8)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation](https://mlsanity.com/paper/2002.02126/) (2020)
- [Neural Collaborative Filtering](https://mlsanity.com/paper/1708.05031/) (2017)
- [Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration](https://mlsanity.com/paper/1910.12656/) (2019)
- [Calibrating Deep Neural Networks using Focal Loss](https://mlsanity.com/paper/2002.09437/) (2020)
- [Recommendations as Treatments: Debiasing Learning and Evaluation](https://mlsanity.com/paper/1602.05352/) (2016)
- [Unbiased Learning-to-Rank with Biased Feedback](https://mlsanity.com/paper/1608.04468/) (2016)
- [Intra Order-preserving Functions for Calibration of Multi-Class Neural Networks](https://mlsanity.com/paper/2003.06820/) (2020)

## Built on by (most cited first, 15 of 15)

- [Uncertainty Quantification and Decomposition for LLM-based Recommendation](https://mlsanity.com/paper/2501.17630/) (2025)
- [Uncertainty Quantification for Fairness in Two-Stage Recommender Systems](https://mlsanity.com/paper/2205.15436/) (2022)
- [Posterior Probability Matters: Doubly-Adaptive Calibration for Neural Predictions in Online Advertising](https://mlsanity.com/paper/2205.07295/) (2022)
- [Doubly Calibrated Estimator for Recommendation on Data Missing Not At Random](https://mlsanity.com/paper/2403.00817/) (2024)
- [MCGrad: Multicalibration at Web Scale](https://mlsanity.com/paper/2509.19884/) (2025)
- [Deep Ensemble Shape Calibration: Multi-Field Post-hoc Calibration in Online Advertising](https://mlsanity.com/paper/2401.09507/) (2024)
- [Stability and Multigroup Fairness in Ranking with Uncertain Predictions](https://mlsanity.com/paper/2402.09326/) (2024)
- [Confidence-Aware Multi-Field Model Calibration](https://mlsanity.com/paper/2402.17655/) (2024)
- [Calibration Matters: Tackling Maximization Bias in Large-scale Advertising Recommendation Systems](https://mlsanity.com/paper/2205.09809/) (2022)
- [Uncertainty Calibration for Counterfactual Propensity Estimation in Recommendation](https://mlsanity.com/paper/2303.12973/) (2023)
- [Calibrating the Predictions for Top-N Recommendations](https://mlsanity.com/paper/2408.11596/) (2024)
- [Calibration-Disentangled Learning and Relevance-Prioritized Reranking for Calibrated Sequential Recommendation](https://mlsanity.com/paper/2408.02156/) (2024)
- [MCNet: Monotonic Calibration Networks for Expressive Uncertainty Calibration in Online Advertising](https://mlsanity.com/paper/2503.00334/) (2025)
- [Rethinking Evaluation Metric for Probability Estimation Models Using Esports Data](https://mlsanity.com/paper/2309.06248/) (2023)
- [Calibrated Preference Learning: The Case of Label Ranking](https://mlsanity.com/paper/2605.30447/) (2026)

## Abstract (arXiv)

For personalized ranking models, the well-calibrated probability of an item being preferred by a user has great practical value. While existing work shows promising results in image classification, probability calibration has not been much explored for personalized ranking. In this paper, we aim to estimate the calibrated probability of how likely a user will prefer an item. We investigate various parametric distributions and propose two parametric calibration methods, namely Gaussian calibration and Gamma calibration. Each proposed method can be seen as a post-processing function that maps the ranking scores of pre-trained models to well-calibrated preference probabilities, without affecting the recommendation performance. We also design the unbiased empirical risk minimization framework that guides the calibration methods to learning of true preference probability from the biased user-item interaction dataset. Extensive evaluations with various personalized ranking models on real-world datasets show that both the proposed calibration methods and the unbiased empirical risk minimization significantly improve the calibration performance.