mlsanityData to 14 Oct 2026

On the Opportunities and Challenges of Offline Reinforcement Learning for Recommender Systems

arXiv 2308.11336 ↗First posted 22 Aug 2023cs.IR cs.AIcited by 10

On the map: Offline RL and imitation learning from data → Decision transformers and sequence modeling

Abstract (arXiv)

Reinforcement learning serves as a potent tool for modeling dynamic user interests within recommender systems, garnering increasing research attention of late. However, a significant drawback persists: its poor data efficiency, stemming from its interactive nature. The training of reinforcement learning-based recommender systems demands expensive online interactions to amass adequate trajectories, essential for agents to learn user preferences. This inefficiency renders reinforcement learning-based recommender systems a formidable undertaking, necessitating the exploration of potential solutions. Recent strides in offline reinforcement learning present a new perspective. Offline reinforcement learning empowers agents to glean insights from offline datasets and deploy learned policies in online settings. Given that recommender systems possess extensive offline datasets, the framework of offline reinforcement learning aligns seamlessly. Despite being a burgeoning field, works centered on recommender systems utilizing offline reinforcement learning remain limited. This survey aims to introduce and delve into offline reinforcement learning within recommender systems, offering an inclusive review of existing literature in this domain. Furthermore, we strive to underscore prevalent challenges, opportunities, and future pathways, poised to propel research in this evolving field.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data. Builds on 46 of its 78 references that are in the map. First posted on arXiv 22 Aug 2023.

Builds on

46 of 78 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Causal Decision Transformer for Recommender Systems via Offline Reinforcement Learning17 Apr
  2. 2022Offline Reinforcement Learning for Mobile Notifications4 Feb

Other references · 44

  1. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  2. 2015Trust Region Policy Optimizationcited by 2,580
  3. 2023A Survey of Large Language Modelscited by 2,576
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  5. 2019BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformercited by 964
  6. 2020Graph Neural Networks in Recommender Systems: A Surveycited by 494
  7. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 339
  8. 2017Deep Reinforcement Learning: An Overviewcited by 333
  9. 2017DeepPath: A Reinforcement Learning Method for Knowledge Graph Reasoningcited by 266
  10. 2020Bias and Debias in Recommender System: A Survey and Future Directionscited by 240
  11. 2018Explainable Recommendation: A Survey and New Perspectivescited by 210
  12. 2023Grounding Large Language Models in Interactive Environments with Online Reinforcement Learningcited by 195
32 more
  1. 2021Reinforcement learning based recommender systems: A surveycited by 188
  2. 2023Guiding Pretraining in Reinforcement Learning with Large Language Modelscited by 187
  3. 2018Top-K Off-Policy Correction for a REINFORCE Recommender Systemcited by 182
  4. 2023Recommendation as Instruction Following: A Large Language Model Empowered Recommendation Approachcited by 168
  5. 2017Tactics of Adversarial Attack on Deep Reinforcement Learning Agentscited by 166
  6. 2018Graph Convolutional Reinforcement Learningcited by 152
  7. 2017Deep Learning based Recommender System: A Survey and New Perspectivescited by 136
  8. 2016Off-policy evaluation for slate recommendationcited by 103
  9. 2018Deep Reinforcement Learning for Page-wise Recommendationscited by 100
  10. 2020Explainability in Deep Reinforcement Learningcited by 97
  11. 2022Robust Reinforcement Learning using Offline Datacited by 96
  12. 2018Virtual-Taobao: Virtualizing Real-world Online Retail Environment for Reinforcement Learningcited by 80
  13. 2021Corruption-Robust Offline Reinforcement Learningcited by 51
  14. 2022CIRS: Bursting Filter Bubbles by Counterfactual Interactive Recommender Systemcited by 38
  15. 2023Alleviating Matthew Effect of Offline Reinforcement Learning in Interactive Recommendationcited by 38
  16. 2021Instrumental Variable Value Iteration for Causal Offline Reinforcement Learningcited by 35
  17. 2023Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learningcited by 27
  18. 2018Policy Gradients for Contextual Recommendationscited by 21
  19. 2020BLOB : A Probabilistic Model for Recommendation that Combines Organic and Bandit Signalscited by 19
  20. 2022Offline Reinforcement Learning with Causal Structured World Modelscited by 19
  21. 2020Knowledge-guided Deep Reinforcement Learning for Interactive Recommendationcited by 18
  22. 2021Improving Zero-shot Generalization in Offline Reinforcement Learning using Generalized Similarity Functionscited by 17
  23. 2022CausPref: Causal Preference Learning for Out-of-Distribution Recommendationcited by 17
  24. 2020Debiased Off-Policy Evaluation for Recommendation Systemscited by 9
  25. 2020Adversarial Attacks and Detection on Reinforcement Learning-Based Interactive Recommender Systemscited by 9
  26. 2022Multi-Task Off-Policy Learning from Bandit Feedbackcited by 9
  27. 2020Learning to Collaborate in Multi-Module Recommendation via Multi-Agent Reinforcement Learning without Communicationcited by 8
  28. 2019Representation Learning on Graphs: A Reinforcement Learning Applicationcited by 5
  29. 2020Offline Meta-level Model-based Reinforcement Learning Approach for Cold-Start Recommendationcited by 5
  30. 2022Intrinsically Motivated Reinforcement Learning based Recommendation with Counterfactual Data Augmentationcited by 5
  31. 2023Offline Evaluation for Reinforcement Learning-based Recommendation: A Critical Issue and Some Alternativescited by 3
  32. 2023Integrating Offline Reinforcement Learning with Transformers for Sequential Recommendationcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 2.

10 papers.

  1. 2025Offline Learning for Combinatorial Multi-armed Banditscited by 5
  2. 2025Policy-Guided Causal State Representation for Offline Reinforcement Learning Recommendationcited by 2
  3. 2025LightKG: Efficient Knowledge-Aware Recommendations with Simplified GNN Architecturecited by 1
  4. 2025Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatismcited by 1
  5. 2026Long-term User Engagement Optimization through Model-agnostic Downstream Rewards Learningcited by 1
  6. 2026TOUR: A Trajectory-Level Unlearning Benchmark for Offline Reinforcement Learningcited by 1
  7. 2025Offline Reinforcement Learning for Microgrid Voltage Regulationcited by 0
  8. 2025Beyond the Proxy: Trajectory-Distilled Guidance for Offline GFlowNet Trainingcited by 0
  9. 2025GrowthHacker: Automated Off-Policy Evaluation Optimization Using Code-Modifying LLM Agentscited by 0
  10. 2025Comparative Analysis of Neural Retriever-Reranker Pipelines for Retrieval-Augmented Generation over Knowledge Graphs in E-commerce Applicationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023A General Offline Reinforcement Learning Framework for Interactive Recommendation
  2. 2025Reward Balancing Revisited: Enhancing Offline Reinforcement Learning for Recommender Systems
  3. 2025DARLR: Dual-Agent Offline Reinforcement Learning for Recommender Systems with Dynamic Reward
  4. 2021Compressive Features in Offline Reinforcement Learning for Recommender Systems
  5. 2024ROLeR: Effective Reward Shaping in Offline Reinforcement Learning for Recommender Systems
  6. 2025Energy-Guided Diffusion Sampling for Long-Term User Behavior Prediction in Reinforcement Learning-based Recommendation
  7. 2021Accelerating Offline Reinforcement Learning Application in Real-Time Bidding and Recommendation: Potential Use of Simulation
  8. 2024EasyRL4Rec: An Easy-to-use Library for Reinforcement Learning Based Recommender Systems