Uncertainty Calibration for Counterfactual Propensity Estimation in Recommendation
Not on the map. Why?
Abstract (arXiv)
Post-click conversion rate (CVR) is a reliable indicator of online customers' preferences, making it crucial for developing recommender systems. A major challenge in predicting CVR is severe selection bias, arising from users' inherent self-selection behavior and the system's item selection process. To mitigate this issue, the inverse propensity score (IPS) is employed to weight the prediction error of each observed instance. However, current propensity score estimations are unreliable due to the lack of a quality measure. To address this, we evaluate the quality of propensity scores from the perspective of uncertainty calibration, proposing the use of Expected Calibration Error (ECE) as a measure of propensity-score quality, which quantifies the extent to which predicted probabilities are overconfident by assessing the difference between predicted probabilities and actual observed frequencies. Miscalibrated propensity scores can lead to distorted IPS weights, thereby compromising the debiasing process in CVR prediction. In this paper, we introduce a model-agnostic calibration framework for propensity-based debiasing of CVR predictions. Theoretical analysis on bias and generalization bounds demonstrates the superiority of calibrated propensity estimates over uncalibrated ones. Experiments conducted on the Coat, Yahoo and KuaiRand datasets show improved uncertainty calibration, as evidenced by lower ECE values, leading to enhanced CVR prediction outcomes.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 44 of its 107 references that are in the map. First posted on arXiv 23 Mar 2023.
Builds on
44 of 107 references.
References · 44
- 2017On Calibration of Modern Neural Networkscited by 4,860
- 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
- 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
- 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
- 2017Neural Collaborative Filteringcited by 1,108
- 2020A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challengescited by 813
- 2016Wide & Deep Learning for Recommender Systemscited by 745
- 2017DeepFM: A Factorization-Machine based Neural Network for CTR Predictioncited by 673
- 2020Bayesian Deep Learning and a Probabilistic Perspective of Generalizationcited by 480
- 2018Accurate Uncertainties for Deep Learning Using Calibrated Regressioncited by 417
- 2017Deep & Cross Network for Ad Click Predictionscited by 392
- 2020BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learningcited by 368
32 more
- 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
- 2019Verified Uncertainty Calibrationcited by 307
- 2016Recommendations as Treatments: Debiasing Learning and Evaluationcited by 246
- 2020Bias and Debias in Recommender System: A Survey and Future Directionscited by 239
- 2018Entire Space Multi-Task Model: An Effective Approach for Estimating Post-Click Conversion Ratecited by 172
- 2019Evaluating model calibration in classificationcited by 165
- 2018Offline A/B testing for Recommender Systemscited by 94
- 2017Causal Embeddings for Recommendationcited by 79
- 2021AutoDebias: Learning to Debias for Recommendationcited by 68
- 2019Uncertainty Quantification with Statistical Guarantees in End-to-End Autonomous Driving Controlcited by 60
- 2022ESCM2: Entire Space Counterfactual Multi-Task Model for Post-Click Conversion Rate Estimationcited by 47
- 2019Large-scale Causal Approaches to Debiasing Post-click Conversion Rate Estimation with Multi-task Learningcited by 39
- 2021Don't Just Blame Over-parametrization for Over-confidence: Theoretical Analysis of Calibration in Binary Classificationcited by 39
- 2022Calibrated One-class Classification for Unsupervised Time Series Anomaly Detectioncited by 39
- 2020Calibrated Reliable Regression using Maximum Mean Discrepancycited by 36
- 2020Information Theoretic Counterfactual Learning from Missing-Not-At-Random Feedbackcited by 30
- 2019Asymmetric Tri-training for Debiasing Missing-Not-At-Random Explicit Feedbackcited by 28
- 2021Enhanced Doubly Robust Learning for Debiasing Post-click Conversion Rate Estimationcited by 28
- 2021Popularity Bias Is Not Always Evil: Disentangling Benign and Harmful Bias for Recommendationcited by 23
- 2020Unbiased Learning for the Causal Effect of Recommendationcited by 18
- 2022A Generalized Doubly Robust Learning Framework for Debiasing Post-Click Conversion Rate Predictioncited by 18
- 2020Adversarial Counterfactual Learning and Evaluation for Recommender Systemcited by 16
- 2021Obtaining Calibrated Probabilities with Personalized Ranking Modelscited by 15
- 2022UKD: Debiasing Conversion Rate Estimation via Uncertainty-regularized Knowledge Distillationcited by 14
- 2020Modeling Perception Errors towards Robust Decision Making in Autonomous Vehiclescited by 11
- 2016Multi-behavioral Sequential Prediction with Recurrent Log-bilinear Modelcited by 9
- 2022Posterior Probability Matters: Doubly-Adaptive Calibration for Neural Predictions in Online Advertisingcited by 8
- 2020SamWalker++: recommendation with informative sampling strategycited by 5
- 2022RMT-Net: Reject-aware Multi-Task Network for Modeling Missing-not-at-random Data in Financial Credit Scoringcited by 5
- 2014Big Learning with Bayesian Methodscited by 4
- 2023Dually Enhanced Propensity Score Estimation in Sequential Recommendationcited by 3
- 2018Uncertainty in the Design Stage of Two-Stage Bayesian Propensity Score Analysiscited by 1
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.