mlsanityData to 7 Oct 2026

Value Penalized Q-Learning for Recommender Systems

arXiv 2110.07923 ↗First posted 15 Oct 2021cs.LG cs.AIcited by 4

On the map: Offline RL and imitation learning from data → Decision transformers and sequence modeling

Abstract (arXiv)

Scaling reinforcement learning (RL) to recommender systems (RS) is promising since maximizing the expected cumulative rewards for RL agents meets the objective of RS, i.e., improving customers' long-term satisfaction. A key approach to this goal is offline RL, which aims to learn policies from logged data. However, the high-dimensional action space and the non-stationary dynamics in commercial RS intensify distributional shift issues, making it challenging to apply offline RL methods to RS. To alleviate the action distribution shift problem in extracting RL policy from static trajectories, we propose Value Penalized Q-learning (VPQ), an uncertainty-based offline RL algorithm. It penalizes the unstable Q-values in the regression target by uncertainty-aware weights, without the need to estimate the behavior policy, suitable for RS with a large number of items. We derive the penalty weights from the variances across an ensemble of Q-functions. To alleviate distributional shift issues at test time, we further introduce the critic framework to integrate the proposed method with classic RS models. Extensive experiments conducted on two real-world datasets show that the proposed method could serve as a gain plugin for existing RS models.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data. Builds on 30 of its 47 references that are in the map. First posted on arXiv 15 Oct 2021.

Builds on

30 of 47 references.

References · 30

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  2. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  3. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  5. 2018Self-Attentive Sequential Recommendationcited by 1,169
  6. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  7. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  8. 2015Session-based Recommendations with Recurrent Neural Networkscited by 685
  9. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  10. 2016Deep Exploration via Bootstrapped DQNcited by 518
  11. 2018Personalized Top-N Sequential Recommendation via Convolutional Sequence Embeddingcited by 473
  12. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
18 more
  1. 2020Is Pessimism Provably Efficient for Offline RL?cited by 362
  2. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  3. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularizationcited by 237
  4. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 211
  5. 2021Reinforcement learning based recommender systems: A surveycited by 187
  6. 2018Top-K Off-Policy Correction for a REINFORCE Recommender Systemcited by 181
  7. 2015Deep Reinforcement Learning in Large Discrete Action Spacescited by 158
  8. 2019Mastering Complex Control in MOBA Games with Deep Reinforcement Learningcited by 150
  9. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learningcited by 125
  10. 2018A Simple Convolutional Generative Network for Next Item Recommendationcited by 120
  11. 2016Off-policy evaluation for slate recommendationcited by 102
  12. 2018Deep Reinforcement Learning for Page-wise Recommendationscited by 99
  13. 2020The Importance of Pessimism in Fixed-Dataset Policy Optimizationcited by 99
  14. 2018Recommendations with Negative Feedback via Pairwise Deep Reinforcement Learningcited by 71
  15. 2020Self-Supervised Reinforcement Learning for Recommender Systemscited by 49
  16. 2020Cost-Sensitive Portfolio Selection via Deep Reinforcement Learningcited by 21
  17. 2019Exact-K Recommendation via Maximal Clique Optimizationcited by 14
  18. 2015Deep Reinforcement Learning with Attention for Slate Markov Decision Processes with High-Dimensional States and Actionscited by 13

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 1.

4 papers.

  1. 2024Towards an Information Theoretic Framework of Context-Based Offline Meta-Reinforcement Learningcited by 16
  2. 2023How to Fine-tune the Model: Unified Model Shift and Model Bias Policy Optimizationcited by 6
  3. 2023Deploying Offline Reinforcement Learning with Human Feedbackcited by 2
  4. 2023Model-free Reinforcement Learning with Stochastic Reward Stabilization for Recommender Systemscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Reward Balancing Revisited: Enhancing Offline Reinforcement Learning for Recommender Systems
  2. 2024ROLeR: Effective Reward Shaping in Offline Reinforcement Learning for Recommender Systems
  3. 2025DARLR: Dual-Agent Offline Reinforcement Learning for Recommender Systems with Dynamic Reward
  4. 2021Compressive Features in Offline Reinforcement Learning for Recommender Systems
  5. 2021Reinforcement Learning with a Disentangled Universal Value Function for Item Recommendation
  6. 2021RL4RS: A Real-World Dataset for Reinforcement Learning based Recommender System
  7. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble
  8. 2023Robust Reinforcement Learning Objectives for Sequential Recommender Systems