mlsanityData to 7 Oct 2026

Top-K Off-Policy Correction for a REINFORCE Recommender System

arXiv 1812.02353 ↗First posted 6 Dec 2018cs.LG cs.IR stat.MLcited by 181

Not on the map. Why?

Cited by 22 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Industrial recommender systems deal with extremely large action spaces -- many millions of items to recommend. Moreover, they need to serve billions of users, who are unique at any point in time, making a complex user state space. Luckily, huge quantities of logged implicit feedback (e.g., user clicks, dwell time) are available for learning. Learning from the logged feedback is however subject to biases caused by only observing feedback on recommendations selected by the previous versions of the recommender. In this work, we present a general recipe of addressing such biases in a production top-K recommender system at Youtube, built with a policy-gradient-based algorithm, i.e. REINFORCE. The contributions of the paper are: (1) scaling REINFORCE to a production recommender system with an action space on the orders of millions; (2) applying off-policy correction to address data biases in learning from logged feedback collected from multiple behavior policies; (3) proposing a novel top-K off-policy correction to account for our policy recommending multiple items at a time; (4) showcasing the value of exploration. We demonstrate the efficacy of our approaches through a series of simulations and multiple live experiments on Youtube.

Key facts. Cited by 181 papers in the map (data through 7 Oct 2026). First posted on arXiv 6 Dec 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 34; 2022: 36; 2023: 37; 2024: 27; 2025: 27; 2026: 19.

181 papers.

Sort:

By area:

  1. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  2. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coveragecited by 160
  3. 2021Advances and Challenges in Conversational Recommender Systems: A Surveycited by 118
  4. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?cited by 86
  5. 2022PinnerFormer: Sequence Modeling for User Representation at Pinterestcited by 57
  6. 2021Zero-Shot Recommender Systemscited by 45
  7. 2024Improving Large Language Models via Fine-grained Reinforcement Learning with Minimum Editing Constraintcited by 39
  8. 2021Nearly Horizon-Free Offline Reinforcement Learningcited by 36
  9. 2022Modeling Content Creator Incentives on Algorithm-Curated Platformscited by 35
  10. 2025PLUM: Adapting Pre-trained Language Models for Industrial-scale Generative Recommendationscited by 27
  11. 2021Reward Poisoning in Reinforcement Learning: Attacks Against Unknown Learners in Unknown Environmentscited by 26
  12. 2021Online Bootstrap Inference For Policy Evaluation in Reinforcement Learningcited by 26
38 more
  1. 2023Multi-Task Recommendations with Reinforcement Learningcited by 26
  2. 2022Preference Dynamics Under Personalized Recommendationscited by 25
  3. 2021Showing Your Offline Reinforcement Learning Work: Online Evaluation Budget Matterscited by 24
  4. 2023Large Language Models for User Interest Journeyscited by 24
  5. 2021RecSim NG: Toward Principled Uncertainty Modeling for Recommender Ecosystemscited by 21
  6. 2021Measuring Recommender System Effects with Simulated Userscited by 20
  7. 2021A Survey of Deep Reinforcement Learning in Recommender Systems: A Systematic Review and Future Directionscited by 19
  8. 2026Self-Evolving Recommendation System: End-To-End Autonomous Model Optimization With LLM Agentscited by 19
  9. 2023A General Offline Reinforcement Learning Framework for Interactive Recommendationcited by 18
  10. 2021Accelerating Offline Reinforcement Learning Application in Real-Time Bidding and Recommendation: Potential Use of Simulationcited by 16
  11. 2023Two-Stage Constrained Actor-Critic for Short Video Recommendationcited by 16
  12. 2023Reinforcement Unlearningcited by 16
  13. 2024Logarithmic Smoothing for Pessimistic Off-Policy Evaluation, Selection and Learningcited by 16
  14. 2024LLMs for User Interest Exploration in Large-scale Recommendation Systemscited by 15
  15. 2021On component interactions in two-stage recommender systemscited by 14
  16. 2021Deep Exploration for Recommendation Systemscited by 14
  17. 2021Choosing the Best of Both Worlds: Diverse and Novel Recommendations through Multi-Objective Reinforcement Learningcited by 14
  18. 2021Control Variates for Slate Off-Policy Evaluationcited by 13
  19. 2022On Gap-dependent Bounds for Offline Reinforcement Learningcited by 13
  20. 2023On (Normalised) Discounted Cumulative Gain as an Off-Policy Evaluation Metric for Top-n Recommendationcited by 13
  21. 2024Optimal Baseline Corrections for Off-Policy Contextual Banditscited by 13
  22. 2023Generative Slate Recommendation with Reinforcement Learningcited by 12
  23. 2025LapSum -- One Method to Differentiate Them All: Ranking, Sorting and Top-k Selectioncited by 12
  24. 2023Ad-load Balancing via Off-policy Learning in a Content Marketplacecited by 11
  25. 2024SUBER: An RL Environment with Simulated Human Behavior for Recommender Systemscited by 11
  26. 2021Recommendation Fairness: From Static to Dynamiccited by 10
  27. 2021SelectAugment: Hierarchical Deterministic Sample Selection for Data Augmentationcited by 10
  28. 2022Uncertainty Quantification for Fairness in Two-Stage Recommender Systemscited by 10
  29. 2022ResAct: Reinforcing Long-term Engagement in Sequential Recommendation with Residual Actorcited by 10
  30. 2023On the Opportunities and Challenges of Offline Reinforcement Learning for Recommender Systemscited by 10
  31. 2024Multi-Objective Recommendation via Multivariate Policy Learningcited by 10
  32. 2021Towards Content Provider Aware Recommender Systems: A Simulation Study on the Interplay between User and Provider Utilitiescited by 9
  33. 2021Reinforcement Learning to Optimize Lifetime Value in Cold-Start Recommendationcited by 9
  34. 2024Delta-rm OPE: Off-Policy Estimation with Pairs of Policiescited by 9
  35. 2021Quantifying Availability and Discovery in Recommender Systems via Stochastic Reachabilitycited by 8
  36. 2021Supervised Advantage Actor-Critic for Recommender Systemscited by 8
  37. 2022Offline Reinforcement Learning for Mobile Notificationscited by 8
  38. 2022Local Policy Improvement for Recommender Systemscited by 8

50 most cited of 181 All 181 (JSON)