# Self-Supervised Reinforcement Learning for Recommender Systems

arXiv: https://arxiv.org/abs/2006.05779 · HTML: https://mlsanity.com/paper/2006.05779/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Jun 2020.

## Built on by (most cited first, 30 of 49)

- [Rec-R1: Bridging Generative Large Language Models and User-Centric Recommendation Systems via Reinforcement Learning](https://mlsanity.com/paper/2503.24289/) (2025)
- [Double-Scale Self-Supervised Hypergraph Learning for Group Recommendation](https://mlsanity.com/paper/2109.04200/) (2021)
- [Showing Your Offline Reinforcement Learning Work: Online Evaluation Budget Matters](https://mlsanity.com/paper/2110.04156/) (2021)
- [Lightweight Self-Attentive Sequential Recommendation](https://mlsanity.com/paper/2108.11333/) (2021)
- [A Survey of Deep Reinforcement Learning in Recommender Systems: A Systematic Review and Future Directions](https://mlsanity.com/paper/2109.03540/) (2021)
- [A General Offline Reinforcement Learning Framework for Interactive Recommendation](https://mlsanity.com/paper/2310.00678/) (2023)
- [A Survey on Deep Reinforcement Learning for Audio-Based Applications](https://mlsanity.com/paper/2101.00240/) (2021)
- [Revisiting Neural Retrieval on Accelerators](https://mlsanity.com/paper/2306.04039/) (2023)
- [Recommendation Fairness: From Static to Dynamic](https://mlsanity.com/paper/2109.03150/) (2021)
- [Local Policy Improvement for Recommender Systems](https://mlsanity.com/paper/2212.11431/) (2022)
- [EasyRL4Rec: An Easy-to-use Library for Reinforcement Learning Based Recommender Systems](https://mlsanity.com/paper/2402.15164/) (2024)
- [Supervised Advantage Actor-Critic for Recommender Systems](https://mlsanity.com/paper/2111.03474/) (2021)
- [Optimizing Audio Recommendations for the Long-Term: A Reinforcement Learning Perspective](https://mlsanity.com/paper/2302.03561/) (2023)
- [PinRec: Unified Generative Retrieval for Pinterest Recommender Systems](https://mlsanity.com/paper/2504.10507/) (2025)
- [Optimizing Long-term Value for Auction-Based Recommender Systems via On-Policy Reinforcement Learning](https://mlsanity.com/paper/2305.13747/) (2023)
- [Value Penalized Q-Learning for Recommender Systems](https://mlsanity.com/paper/2110.07923/) (2021)
- [On the Effectiveness of Unlearning in Session-Based Recommendation](https://mlsanity.com/paper/2312.14447/) (2023)
- [ROLeR: Effective Reward Shaping in Offline Reinforcement Learning for Recommender Systems](https://mlsanity.com/paper/2407.13163/) (2024)
- [Towards Off-Policy Reinforcement Learning for Ranking Policies with Human Feedback](https://mlsanity.com/paper/2401.08959/) (2024)
- [DARLR: Dual-Agent Offline Reinforcement Learning for Recommender Systems with Dynamic Reward](https://mlsanity.com/paper/2505.07257/) (2025)
- [Sequence Adaptation via Reinforcement Learning in Recommender Systems](https://mlsanity.com/paper/2108.01442/) (2021)
- [Hierarchical Reinforcement Learning for Modeling User Novelty-Seeking Intent in Recommender Systems](https://mlsanity.com/paper/2306.01476/) (2023)
- [Model-free Reinforcement Learning with Stochastic Reward Stabilization for Recommender Systems](https://mlsanity.com/paper/2308.13246/) (2023)
- [Model-enhanced Contrastive Reinforcement Learning for Sequential Recommendation](https://mlsanity.com/paper/2310.16566/) (2023)
- [Optimal Policy Learning with Observational Data in Multi-Action Scenarios: Estimation, Risk Preference, and Potential Failures](https://mlsanity.com/paper/2403.20250/) (2024)
- [Comparative Performance of Collaborative Bandit Algorithms: Effect of Sparsity and Exploration Intensity](https://mlsanity.com/paper/2410.12086/) (2024)
- [Goal-Conditioned Supervised Learning for Multi-Objective Recommendation](https://mlsanity.com/paper/2412.08911/) (2024)
- [Multi-Task Learning for User Engagement and Adoption in Live Video Streaming Events](https://mlsanity.com/paper/2106.10305/) (2021)
- [MultiHead MultiModal Deep Interest Recommendation Network](https://mlsanity.com/paper/2110.10205/) (2021)
- [Fine-Grained Session Recommendations in E-commerce using Deep Reinforcement Learning](https://mlsanity.com/paper/2210.15451/) (2022)

## Abstract (arXiv)

In session-based or sequential recommendation, it is important to consider a number of factors like long-term user engagement, multiple types of user-item interactions such as clicks, purchases etc. The current state-of-the-art supervised approaches fail to model them appropriately. Casting sequential recommendation task as a reinforcement learning (RL) problem is a promising direction. A major component of RL approaches is to train the agent through interactions with the environment. However, it is often problematic to train a recommender in an on-line fashion due to the requirement to expose users to irrelevant recommendations. As a result, learning the policy from logged implicit feedback is of vital importance, which is challenging due to the pure off-policy setting and lack of negative rewards (feedback). In this paper, we propose self-supervised reinforcement learning for sequential recommendation tasks. Our approach augments standard recommendation models with two output layers: one for self-supervised learning and the other for RL. The RL part acts as a regularizer to drive the supervised layer focusing on specific rewards(e.g., recommending items which may lead to purchases rather than clicks) while the self-supervised layer with cross-entropy loss provides strong gradient signals for parameter updates. Based on such an approach, we propose two frameworks namely Self-Supervised Q-learning(SQN) and Self-Supervised Actor-Critic(SAC). We integrate the proposed frameworks with four state-of-the-art recommendation models. Experimental results on two real-world datasets demonstrate the effectiveness of our approach.