# A Survey of Deep Reinforcement Learning in Recommender Systems: A Systematic Review and Future Directions

arXiv: https://arxiv.org/abs/2109.03540 · HTML: https://mlsanity.com/paper/2109.03540/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 73 of its 132 references that are in the map.
- First posted on arXiv 8 Sep 2021.

## Builds on (most cited first, 30 of 73)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017)
- [Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks](https://mlsanity.com/paper/1703.10593/) (2017)
- [A Brief Survey of Deep Reinforcement Learning](https://mlsanity.com/paper/1708.05866/) (2017)
- [Personalized Top-N Sequential Recommendation via Convolutional Sequence Embedding](https://mlsanity.com/paper/1809.07426/) (2018)
- [Time-Contrastive Networks: Self-Supervised Learning from Video](https://mlsanity.com/paper/1704.06888/) (2017)
- [Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey](https://mlsanity.com/paper/2009.13303/) (2020)
- [Reinforcement Learning with Unsupervised Auxiliary Tasks](https://mlsanity.com/paper/1611.05397/) (2016)
- [Learning to reinforcement learn](https://mlsanity.com/paper/1611.05763/) (2016)
- [Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation](https://mlsanity.com/paper/1604.06057/) (2016)
- [Recommendations as Treatments: Debiasing Learning and Evaluation](https://mlsanity.com/paper/1602.05352/) (2016)
- [Selective Experience Replay for Lifelong Learning](https://mlsanity.com/paper/1802.10269/) (2018)
- [Bias and Debias in Recommender System: A Survey and Future Directions](https://mlsanity.com/paper/2010.03240/) (2020)
- [Adversarial Policies: Attacking Deep Reinforcement Learning](https://mlsanity.com/paper/1905.10615/) (2019)
- [Explainable Recommendation: A Survey and New Perspectives](https://mlsanity.com/paper/1804.11192/) (2018)
- [Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning](https://mlsanity.com/paper/1809.02925/) (2018)
- [Reinforcement learning based recommender systems: A survey](https://mlsanity.com/paper/2101.06286/) (2021)
- [Top-K Off-Policy Correction for a REINFORCE Recommender System](https://mlsanity.com/paper/1812.02353/) (2018)
- [Explainable Reinforcement Learning Through a Causal Lens](https://mlsanity.com/paper/1905.10958/) (2019)
- [Tactics of Adversarial Attack on Deep Reinforcement Learning Agents](https://mlsanity.com/paper/1703.06748/) (2017)

## Built on by (most cited first, 21 of 21)

- [A Survey of Sim-to-Real Methods in RL: Progress, Prospects and Challenges with Foundation Models](https://mlsanity.com/paper/2502.13187/) (2025)
- [Searching for High-Value Molecules Using Reinforcement Learning and Transformers](https://mlsanity.com/paper/2310.02902/) (2023)
- [Reinforcement Learning for Generative AI: A Survey](https://mlsanity.com/paper/2308.14328/) (2023)
- [Synthetic Data-Based Simulators for Recommender Systems: A Survey](https://mlsanity.com/paper/2206.11338/) (2022)
- [Evolutionary Reinforcement Learning: A Systematic Review and Future Directions](https://mlsanity.com/paper/2402.13296/) (2024)
- [Pessimistic Model Selection for Offline Deep Reinforcement Learning](https://mlsanity.com/paper/2111.14346/) (2021)
- [Marketing Budget Allocation with Offline Constrained Deep Reinforcement Learning](https://mlsanity.com/paper/2309.02669/) (2023)
- [RecoMind: A Reinforcement Learning Framework for Optimizing In-Session User Satisfaction in Recommendation Systems](https://mlsanity.com/paper/2508.00201/) (2025)
- [Multi-task Offline Reinforcement Learning for Online Advertising in Recommender Systems](https://mlsanity.com/paper/2506.23090/) (2025)
- [Bi-directional personalization reinforcement learning-based architecture with active learning using a multi-model data service for the travel nursing industry](https://mlsanity.com/paper/2304.00006/) (2023)
- [Model-free Reinforcement Learning with Stochastic Reward Stabilization for Recommender Systems](https://mlsanity.com/paper/2308.13246/) (2023)
- [FedSlate:A Federated Deep Reinforcement Learning Recommender System](https://mlsanity.com/paper/2409.14872/) (2024)
- [Toward Debugging Deep Reinforcement Learning Programs with RLExplorer](https://mlsanity.com/paper/2410.04322/) (2024)
- [Statistical Guarantees for Offline Domain Randomization](https://mlsanity.com/paper/2506.10133/) (2025)
- [Fine-Grained Session Recommendations in E-commerce using Deep Reinforcement Learning](https://mlsanity.com/paper/2210.15451/) (2022)
- [Situating Recommender Systems in Practice: Towards Inductive Learning and Incremental Updates](https://mlsanity.com/paper/2211.06365/) (2022)
- [Position Paper: Why the Shooting in the Dark Method Dominates Recommender Systems Practice; A Call to Abandon Anti-Utopian Thinking](https://mlsanity.com/paper/2402.02152/) (2024)
- [Debiased Model-based Interactive Recommendation](https://mlsanity.com/paper/2402.15819/) (2024)
- [An Advantage-based Optimization Method for Reinforcement Learning in Large Action Space](https://mlsanity.com/paper/2412.12605/) (2024)
- [Developing AI Agents with Simulated Data: Why, what, and how?](https://mlsanity.com/paper/2602.15816/) (2026)
- [Reinforcement Learning with Conformal Action Sets: An Application to Sequential Recommendation](https://mlsanity.com/paper/2610.08743/) (2026)

## Abstract (arXiv)

In light of the emergence of deep reinforcement learning (DRL) in recommender systems research and several fruitful results in recent years, this survey aims to provide a timely and comprehensive overview of the recent trends of deep reinforcement learning in recommender systems. We start with the motivation of applying DRL in recommender systems. Then, we provide a taxonomy of current DRL-based recommender systems and a summary of existing methods. We discuss emerging topics and open issues, and provide our perspective on advancing the domain. This survey serves as introductory material for readers from academia and industry into the topic and identifies notable opportunities for further research.