# Robust Reinforcement Learning Objectives for Sequential Recommender Systems

arXiv: https://arxiv.org/abs/2305.18820 · HTML: https://mlsanity.com/paper/2305.18820/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 34 of its 52 references that are in the map.
- First posted on arXiv 30 May 2023.

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Self-Attentive Sequential Recommendation](https://mlsanity.com/paper/1808.09781/) (2018)
- [Graph Convolutional Neural Networks for Web-Scale Recommender Systems](https://mlsanity.com/paper/1806.01973/) (2018)
- [BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer](https://mlsanity.com/paper/1904.06690/) (2019)
- [Personalized Top-N Sequential Recommendation via Convolutional Sequence Embedding](https://mlsanity.com/paper/1809.07426/) (2018)
- [S^3-Rec: Self-Supervised Learning for Sequential Recommendation with Mutual Information Maximization](https://mlsanity.com/paper/2008.07873/) (2020)
- [Reinforcement learning based recommender systems: A survey](https://mlsanity.com/paper/2101.06286/) (2021)
- [Top-K Off-Policy Correction for a REINFORCE Recommender System](https://mlsanity.com/paper/1812.02353/) (2018)
- [Behavior Sequence Transformer for E-commerce Recommendation in Alibaba](https://mlsanity.com/paper/1905.06874/) (2019)
- [Contrastive Learning for Representation Degeneration Problem in Sequential Recommendation](https://mlsanity.com/paper/2110.05730/) (2021)
- [A Simple Convolutional Generative Network for Next Item Recommendation](https://mlsanity.com/paper/1808.05163/) (2018)
- [GPT4Rec: A Generative Framework for Personalized Recommendation and User Interests Interpretation](https://mlsanity.com/paper/2304.03879/) (2023)
- [Sequential Recommendation with Graph Neural Networks](https://mlsanity.com/paper/2106.14226/) (2021)
- [Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoders](https://mlsanity.com/paper/2104.08027/) (2021)
- [Recommendations with Negative Feedback via Pairwise Deep Reinforcement Learning](https://mlsanity.com/paper/1802.06501/) (2018)
- [Generative Adversarial User Model for Reinforcement Learning Based Recommendation System](https://mlsanity.com/paper/1812.10613/) (2018)
- [Scaleformer: Iterative Multi-scale Refining Transformers for Time Series Forecasting](https://mlsanity.com/paper/2206.04038/) (2022)
- [Generating Personalized Recipes from Historical User Preferences](https://mlsanity.com/paper/1909.00105/) (2019)
- [Self-Supervised Reinforcement Learning for Recommender Systems](https://mlsanity.com/paper/2006.05779/) (2020)
- [A Loss Curvature Perspective on Training Instability in Deep Learning](https://mlsanity.com/paper/2110.04369/) (2021)
- [Model-Based Reinforcement Learning with Adversarial Training for Online Recommendation](https://mlsanity.com/paper/1911.03845/) (2019)
- [Locality and compositionality in zero-shot learning](https://mlsanity.com/paper/1912.12179/) (2019)
- [Contrastive Learning for Recommender System](https://mlsanity.com/paper/2101.01317/) (2021)

## Abstract (arXiv)

Attention-based sequential recommendation methods have shown promise in accurately capturing users' evolving interests from their past interactions. Recent research has also explored the integration of reinforcement learning (RL) into these models, in addition to generating superior user representations. By framing sequential recommendation as an RL problem with reward signals, we can develop recommender systems that incorporate direct user feedback in the form of rewards, enhancing personalization for users. Nonetheless, employing RL algorithms presents challenges, including off-policy training, expansive combinatorial action spaces, and the scarcity of datasets with sufficient reward signals. Contemporary approaches have attempted to combine RL and sequential modeling, incorporating contrastive-based objectives and negative sampling strategies for training the RL component. In this work, we further emphasize the efficacy of contrastive-based objectives paired with augmentation to address datasets with extended horizons. Additionally, we recognize the potential instability issues that may arise during the application of negative sampling. These challenges primarily stem from the data imbalance prevalent in real-world datasets, which is a common issue in offline RL contexts. Furthermore, we introduce an enhanced methodology aimed at providing a more effective solution to these challenges. Experimental results across several real datasets show our method with increased robustness and state-of-the-art performance.