mlsanityData to 14 Oct 2026

Local Policy Improvement for Recommender Systems

arXiv 2212.11431 ↗First posted 22 Dec 2022cs.LG cs.IRcited by 9

Not on the map. Why?

Abstract (arXiv)

Recommender systems predict what items a user will interact with next, based on their past interactions. The problem is often approached through supervised learning, but recent advancements have shifted towards policy optimization of rewards (e.g., user engagement). One challenge with the latter is policy mismatch: we are only able to train a new policy given data collected from a previously-deployed policy. The conventional way to address this problem is through importance sampling correction, but this comes with practical limitations. We suggest an alternative approach of local policy improvement without off-policy correction. Our method computes and optimizes a lower bound of expected reward of the target policy, which is easy to estimate from data and does not involve density ratios (such as those appearing in importance sampling correction). This local policy improvement paradigm is ideal for recommender systems, as previous policies are typically of decent quality and policies are updated frequently. We provide empirical evidence and practical recipes for applying our technique in a sequential recommendation setting.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 35 of its 58 references that are in the map. First posted on arXiv 22 Dec 2022.

Builds on

35 of 58 references.

References · 35

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2015Trust Region Policy Optimizationcited by 2,580
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  5. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  6. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  7. 2016Variational Inference: A Review for Statisticianscited by 1,472
  8. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  9. 2018Self-Attentive Sequential Recommendationcited by 1,184
  10. 2017Constrained Policy Optimizationcited by 983
  11. 2019BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformercited by 964
  12. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
23 more
  1. 2019Behavior Regularized Offline Reinforcement Learningcited by 662
  2. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  3. 2018Personalized Top-N Sequential Recommendation via Convolutional Sequence Embeddingcited by 476
  4. 2018Variational Autoencoders for Collaborative Filteringcited by 310
  5. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 286
  6. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 279
  7. 2016Recommendations as Treatments: Debiasing Learning and Evaluationcited by 248
  8. 2018Top-K Off-Policy Correction for a REINFORCE Recommender Systemcited by 182
  9. 2021Offline RL Without Off-Policy Evaluationcited by 151
  10. 2018A Simple Convolutional Generative Network for Next Item Recommendationcited by 120
  11. 2018Deep Reinforcement Learning for Page-wise Recommendationscited by 100
  12. 2018Offline A/B testing for Recommender Systemscited by 96
  13. 2019Reinforcement Learning to Optimize Long-term User Engagement in Recommender Systemscited by 90
  14. 2017Causal Embeddings for Recommendationcited by 79
  15. 2018Recommendations with Negative Feedback via Pairwise Deep Reinforcement Learningcited by 71
  16. 2015Modeling User Exposure in Recommendationcited by 69
  17. 2020Off-policy Bandits with Deficient Supportcited by 61
  18. 2020Self-Supervised Reinforcement Learning for Recommender Systemscited by 49
  19. 2015Counterfactual Risk Minimization: Learning from Logged Bandit Feedbackcited by 33
  20. 2021Evaluating the Robustness of Off-Policy Evaluationcited by 32
  21. 2019Imitation-Regularized Offline Learningcited by 17
  22. 2022Rethinking Reinforcement Learning for Recommendation: A Prompt Perspectivecited by 10
  23. 2021Supervised Advantage Actor-Critic for Recommender Systemscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 5; 2026: 3.

9 papers.

  1. 2024POTEC: Off-Policy Learning for Large Action Spaces via Two-Stage Policy Decompositioncited by 13
  2. 2025Offline RL by Reward-Weighted Fine-Tuning for Conversation Optimizationcited by 10
  3. 2025Off-Policy Evaluation and Learning for Matching Marketscited by 2
  4. 2025Optimization of Epsilon-Greedy Explorationcited by 1
  5. 2025Off-Policy Learning in Large Action Spaces: Optimization Matters More Than Estimationcited by 1
  6. 2025Learning to Reason in LLMs by Expectation Maximizationcited by 0
  7. 2026Robust Post-Training for Generative Recommenders: Why Exponential Reward-Weighted SFT Outperforms RLHFcited by 0
  8. 2026On-Policy and Off-Policy Learning for Large Action Spacescited by 0
  9. 2026Efficient Offline Learning of Ranking Policies via Top-k Policy Decompositioncited by 0