# Unbiased Recommender Learning from Missing-Not-At-Random Implicit Feedback

arXiv: https://arxiv.org/abs/1909.03601 · HTML: https://mlsanity.com/paper/1909.03601/

## Key facts

- Cited by 79 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Sep 2019.

## Built on by (most cited first, 30 of 79)

- [AutoDebias: Learning to Debias for Recommendation](https://mlsanity.com/paper/2105.04170/) (2021)
- [Doubly Robust Off-Policy Evaluation for Ranking Policies under the Cascade Behavior Model](https://mlsanity.com/paper/2202.01562/) (2022)
- [Enhanced Doubly Robust Learning for Debiasing Post-click Conversion Rate Estimation](https://mlsanity.com/paper/2105.13623/) (2021)
- [StableDR: Stabilized Doubly Robust Learning for Recommendation on Data Missing Not at Random](https://mlsanity.com/paper/2205.04701/) (2022)
- [TDR-CL: Targeted Doubly Robust Collaborative Learning for Debiased Recommendations](https://mlsanity.com/paper/2203.10258/) (2022)
- [Doubly-Robust Estimation for Correcting Position-Bias in Click Feedback for Unbiased Learning to Rank](https://mlsanity.com/paper/2203.17118/) (2022)
- [A Generalized Doubly Robust Learning Framework for Debiasing Post-Click Conversion Rate Prediction](https://mlsanity.com/paper/2211.06684/) (2022)
- [Balancing Unobserved Confounding with a Few Unbiased Ratings in Debiased Recommendations](https://mlsanity.com/paper/2304.09085/) (2023)
- [A General Offline Reinforcement Learning Framework for Interactive Recommendation](https://mlsanity.com/paper/2310.00678/) (2023)
- [Towards Fair Federated Recommendation Learning: Characterizing the Inter-Dependence of System and Data Heterogeneity](https://mlsanity.com/paper/2206.02633/) (2022)
- [Off-Policy Evaluation of Ranking Policies under Diverse User Behavior](https://mlsanity.com/paper/2306.15098/) (2023)
- [Dual Test-time Training for Out-of-distribution Recommender System](https://mlsanity.com/paper/2407.15620/) (2024)
- [On Missing Labels, Long-tails and Propensities in Extreme Multi-label Classification](https://mlsanity.com/paper/2207.13186/) (2022)
- [Deconfounded Causal Collaborative Filtering](https://mlsanity.com/paper/2110.07122/) (2021)
- [Breaking Feedback Loops in Recommender Systems with Causal Inference](https://mlsanity.com/paper/2207.01616/) (2022)
- [A Survey on Fairness-aware Recommender Systems](https://mlsanity.com/paper/2306.00403/) (2023)
- [Off-Policy Evaluation for Large Action Spaces via Policy Convolution](https://mlsanity.com/paper/2310.15433/) (2023)
- [Causal Collaborative Filtering](https://mlsanity.com/paper/2102.01868/) (2021)
- [Entire Space Counterfactual Learning for Reliable Content Recommendations](https://mlsanity.com/paper/2210.11039/) (2022)
- [Robust Collaborative Filtering to Popularity Distribution Shift](https://mlsanity.com/paper/2310.10696/) (2023)
- [SelfGNN: Self-Supervised Graph Neural Networks for Sequential Recommendation](https://mlsanity.com/paper/2405.20878/) (2024)
- [Multiple Robust Learning for Recommendation](https://mlsanity.com/paper/2207.10796/) (2022)
- [Unbiased Learning to Rank Meets Reality: Lessons from Baidu's Large-Scale Search Dataset](https://mlsanity.com/paper/2404.02543/) (2024)
- [Be Causal: De-biasing Social Network Confounding in Recommendation](https://mlsanity.com/paper/2105.07775/) (2021)
- [Recommender systems based on graph embedding techniques: A comprehensive review](https://mlsanity.com/paper/2109.09587/) (2021)
- [Rethinking Neural vs. Matrix-Factorization Collaborative Filtering: the Theoretical Perspectives](https://mlsanity.com/paper/2110.12141/) (2021)
- [Cross Pairwise Ranking for Unbiased Item Recommendation](https://mlsanity.com/paper/2204.12176/) (2022)
- [Debiased Recommendation with Noisy Feedback](https://mlsanity.com/paper/2406.17182/) (2024)
- [Debiased Explainable Pairwise Ranking from Implicit Feedback](https://mlsanity.com/paper/2107.14768/) (2021)
- [Doubly Calibrated Estimator for Recommendation on Data Missing Not At Random](https://mlsanity.com/paper/2403.00817/) (2024)

## Abstract (arXiv)

Recommender systems widely use implicit feedback such as click data because of its general availability. Although the presence of clicks signals the users' preference to some extent, the lack of such clicks does not necessarily indicate a negative response from the users, as it is possible that the users were not exposed to the items (positive-unlabeled problem). This leads to a difficulty in predicting the users' preferences from implicit feedback. Previous studies addressed the positive-unlabeled problem by uniformly upweighting the loss for the positive feedback data or estimating the confidence of each data having relevance information via the EM-algorithm. However, these methods failed to address the missing-not-at-random problem in which popular or frequently recommended items are more likely to be clicked than other items even if a user does not have a considerable interest in them. To overcome these limitations, we first define an ideal loss function to be optimized to realize recommendations that maximize the relevance and propose an unbiased estimator for the ideal loss. Subsequently, we analyze the variance of the proposed unbiased estimator and further propose a clipped estimator that includes the unbiased estimator as a special case. We demonstrate that the clipped estimator is expected to improve the performance of the recommender system, by considering the bias-variance trade-off. We conduct semi-synthetic and real-world experiments and demonstrate that the proposed method largely outperforms the baselines. In particular, the proposed method works better for rare items that are less frequently observed in the training data. The findings indicate that the proposed method can better achieve the objective of recommending items with the highest relevance.