mlsanityData to 14 Oct 2026

Safely Exploring Novel Actions in Recommender Systems via Deployment-Efficient Policy Learning

arXiv 2510.07635 ↗First posted 9 Oct 2025cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

In many real recommender systems, novel items are added frequently over time. The importance of sufficiently presenting novel actions has widely been acknowledged for improving long-term user engagement. A recent work builds on Off-Policy Learning (OPL), which trains a policy from only logged data, however, the existing methods can be unsafe in the presence of novel actions. Our goal is to develop a framework to enforce exploration of novel actions with a guarantee for safety. To this end, we first develop Safe Off-Policy Policy Gradient (Safe OPG), which is a model-free safe OPL method based on a high confidence off-policy evaluation. In our first experiment, we observe that Safe OPG almost always satisfies a safety requirement, even when existing methods violate it greatly. However, the result also reveals that Safe OPG tends to be too conservative, suggesting a difficult tradeoff between guaranteeing safety and exploring novel actions. To overcome this tradeoff, we also propose a novel framework called Deployment-Efficient Policy Learning for Safe User Exploration, which leverages safety margin and gradually relaxes safety regularization during multiple (not many) deployments. Our framework thus enables exploration of novel actions while guaranteeing safe implementation of recommender systems.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 34 of its 80 references that are in the map. First posted on arXiv 9 Oct 2025.

Builds on

34 of 80 references.

References · 34

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  4. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  5. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  6. 2017Constrained Policy Optimizationcited by 983
  7. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  8. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 337
  9. 2019Batch Policy Learning under Constraintscited by 244
  10. 2018Fairness of Exposure in Rankingscited by 190
  11. 2018Top-K Off-Policy Correction for a REINFORCE Recommender Systemcited by 182
  12. 2019Constrained Reinforcement Learning Has Zero Duality Gapcited by 151
22 more
  1. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 144
  2. 2018More Robust Doubly Robust Off-policy Evaluationcited by 141
  3. 2020Provably Efficient Safe Exploration via Primal-Dual Policy Optimizationcited by 134
  4. 2020CRPO: A New Approach for Safe Reinforcement Learning with Convergence Guaranteecited by 134
  5. 2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
  6. 2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learningcited by 91
  7. 2020An empirical investigation of the challenges of real-world reinforcement learningcited by 77
  8. 2020Open Bandit Dataset and Pipeline: Towards Realistic and Reproducible Off-Policy Evaluationcited by 77
  9. 2020Stochastic Bandits with Linear Constraintscited by 68
  10. 2018Accelerated Primal-Dual Policy Optimization for Safe Reinforcement Learningcited by 62
  11. 2020Off-policy Bandits with Deficient Supportcited by 61
  12. 2019Convergent Policy Optimization for Safe Reinforcement Learningcited by 57
  13. 2022Off-Policy Evaluation for Large Action Spaces via Embeddingscited by 54
  14. 2020PECOS: Prediction for Enormous and Correlated Output Spacescited by 50
  15. 2021Fairness of Exposure in Stochastic Banditscited by 43
  16. 2021Evaluating the Robustness of Off-Policy Evaluationcited by 32
  17. 2022Towards Deployment-Efficient Reinforcement Learning: Lower Bound and Optimalitycited by 24
  18. 2019Imitation-Regularized Offline Learningcited by 17
  19. 2019Lifelong Learning with a Changing Action Setcited by 17
  20. 2021Accelerating Offline Reinforcement Learning Application in Real-Time Bidding and Recommendation: Potential Use of Simulationcited by 16
  21. 2020Managing Diversity in Airbnb Searchcited by 11
  22. 2021Combining Online Learning and Offline Learning for Contextual Bandits with Deficient Supportcited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024UOEP: User-Oriented Exploration Policy for Enhancing Long-Term User Experiences in Recommender Systems
  2. 2025Reward Balancing Revisited: Enhancing Offline Reinforcement Learning for Recommender Systems
  3. 2023On the Opportunities and Challenges of Offline Reinforcement Learning for Recommender Systems
  4. 2026ProRL: Effective Reinforcement Learning for Proactive Recommendation via Rectified Policy Gradient Estimation
  5. 2023A General Offline Reinforcement Learning Framework for Interactive Recommendation
  6. 2026Efficient Offline Learning of Ranking Policies via Top-k Policy Decomposition
  7. 2026Don't Let Bandit Feedback Pull Continual LLM-Recommender Updates Off Target
  8. 2020Offline Meta-level Model-based Reinforcement Learning Approach for Cold-Start Recommendation