mlsanityData to 14 Oct 2026

A General Offline Reinforcement Learning Framework for Interactive Recommendation

arXiv 2310.00678 ↗First posted 1 Oct 2023cs.LG cs.IRcited by 18

On the map: Offline RL and imitation learning from data → Decision transformers and sequence modeling

Abstract (arXiv)

This paper studies the problem of learning interactive recommender systems from logged feedbacks without any exploration in online environments. We address the problem by proposing a general offline reinforcement learning framework for recommendation, which enables maximizing cumulative user rewards without online exploration. Specifically, we first introduce a probabilistic generative model for interactive recommendation, and then propose an effective inference algorithm for discrete and stochastic policy learning based on logged feedbacks. In order to perform offline learning more effectively, we propose five approaches to minimize the distribution mismatch between the logging policy and recommendation policy: support constraints, supervised regularization, policy constraints, dual constraints and reward extrapolation. We conduct extensive experiments on two public real-world datasets, demonstrating that the proposed methods can achieve superior performance over existing supervised learning and reinforcement learning methods for recommendation.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data. Builds on 32 of its 53 references that are in the map. First posted on arXiv 1 Oct 2023.

Builds on

32 of 53 references.

References · 32

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  4. 2016Matching Networks for One Shot Learningcited by 2,255
  5. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  6. 2016Variational Inference: A Review for Statisticianscited by 1,472
  7. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  8. 2018Self-Attentive Sequential Recommendationcited by 1,184
  9. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  10. 2015Session-based Recommendations with Recurrent Neural Networkscited by 695
  11. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  12. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 482
20 more
  1. 2018Personalized Top-N Sequential Recommendation via Convolutional Sequence Embeddingcited by 476
  2. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learningcited by 215
  3. 2018Top-K Off-Policy Correction for a REINFORCE Recommender Systemcited by 182
  4. 2019Soft Actor-Critic for Discrete Action Settingscited by 174
  5. 2018A Simple Convolutional Generative Network for Next Item Recommendationcited by 120
  6. 2018Supervised Reinforcement Learning with Recurrent Neural Network for Dynamic Treatment Recommendationcited by 101
  7. 2018Deep Reinforcement Learning for Page-wise Recommendationscited by 100
  8. 2019Reinforcement Learning to Optimize Long-term User Engagement in Recommender Systemscited by 90
  9. 2019Unbiased Recommender Learning from Missing-Not-At-Random Implicit Feedbackcited by 79
  10. 2018Recommendations with Negative Feedback via Pairwise Deep Reinforcement Learningcited by 71
  11. 2018Latent Space Policies for Hierarchical Reinforcement Learningcited by 67
  12. 2018Generative Adversarial User Model for Reinforcement Learning Based Recommendation Systemcited by 64
  13. 2020Off-policy Bandits with Deficient Supportcited by 61
  14. 2020Self-Supervised Reinforcement Learning for Recommender Systemscited by 49
  15. 2019Off-Policy Policy Gradient with State Distribution Correctioncited by 43
  16. 2020Interactive Recommender System via Knowledge Graph-enhanced Reinforcement Learningcited by 40
  17. 2019Model-Based Reinforcement Learning with Adversarial Training for Online Recommendationcited by 33
  18. 2020Jointly Learning to Recommend and Advertisecited by 27
  19. 2019Exact-K Recommendation via Maximal Clique Optimizationcited by 14
  20. 2019Semi-supervisedly Co-embedding Attributed Networkscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 8; 2025: 5; 2026: 2.

18 papers.

Sort:

By area:

  1. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignmentcited by 51
  2. 2023Simple and Asymmetric Graph Contrastive Learning without Augmentationscited by 38
  3. 2025SimPER: A Minimalist Approach to Preference Alignment without Hyperparameterscited by 25
  4. 2024How to Leverage Demonstration Data in Alignment for Large Language Model? A Self-Imitation Learning Perspectivecited by 14
  5. 2025On a Connection Between Imitation Learning and RLHFcited by 13
  6. 20243M-Diffusion: Latent Multi-Modal Diffusion for Language-Guided Molecular Structure Generationcited by 8
  7. 2025InfoPO: On Mutual Information Maximization for Large Language Model Alignmentcited by 7
  8. 2024RePlay: a Recommendation Framework for Experimentation and Production Usecited by 3
  9. 2024Towards Off-Policy Reinforcement Learning for Ranking Policies with Human Feedbackcited by 2
  10. 2024GeomCLIP: Contrastive Geometry-Text Pre-training for Moleculescited by 2
  11. 2025Multi-Fidelity Hybrid Reinforcement Learning via Information Gain Maximizationcited by 2
  12. 2024Contrastive Diffuser: Planning Towards High Return States via Contrastive Learningcited by 1
6 more
  1. 2024Offline Trajectory Optimization for Offline Reinforcement Learningcited by 1
  2. 2026Inference Time Policy Optimization for Offline RL with Differentiable World Modelscited by 1
  3. 2023A General Neural Causal Model for Interactive Recommendationcited by 0
  4. 2023Adversarial Batch Inverse Reinforcement Learning: Learn to Reward from Imperfect Demonstration for Interactive Recommendationcited by 0
  5. 2025Fine-Tuning Diffusion-Based Recommender Systems via Reinforcement Learning with Reward Function Optimizationcited by 0
  6. 2026Target-Aligned Bellman Backup for Cross-domain Offline Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023On the Opportunities and Challenges of Offline Reinforcement Learning for Recommender Systems
  2. 2020Offline Meta-level Model-based Reinforcement Learning Approach for Cold-Start Recommendation
  3. 2025Reward Balancing Revisited: Enhancing Offline Reinforcement Learning for Recommender Systems
  4. 2022Fast Offline Policy Optimization for Large Scale Recommendation
  5. 2025Energy-Guided Diffusion Sampling for Long-Term User Behavior Prediction in Reinforcement Learning-based Recommendation
  6. 2023Model-enhanced Contrastive Reinforcement Learning for Sequential Recommendation
  7. 2019Learning to Recommend from Sparse Data via Generative User Feedback ↗
  8. 2020Generative Inverse Deep Reinforcement Learning for Online Recommendation