# Model-Based Reinforcement Learning with Adversarial Training for Online Recommendation

arXiv: https://arxiv.org/abs/1911.03845 · HTML: https://mlsanity.com/paper/1911.03845/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Nov 2019.

## Built on by (most cited first, 30 of 33)

- [Advances and Challenges in Conversational Recommender Systems: A Survey](https://mlsanity.com/paper/2101.09459/) (2021)
- [Optimizing DDPM Sampling with Shortcut Fine-Tuning](https://mlsanity.com/paper/2301.13362/) (2023)
- [Mismatched No More: Joint Model-Policy Optimization for Model-Based RL](https://mlsanity.com/paper/2110.02758/) (2021)
- [Estimating and Penalizing Induced Preference Shifts in Recommender Systems](https://mlsanity.com/paper/2204.11966/) (2022)
- [Multi-Task Recommendations with Reinforcement Learning](https://mlsanity.com/paper/2302.03328/) (2023)
- [A Survey of Deep Reinforcement Learning in Recommender Systems: A Systematic Review and Future Directions](https://mlsanity.com/paper/2109.03540/) (2021)
- [A General Offline Reinforcement Learning Framework for Interactive Recommendation](https://mlsanity.com/paper/2310.00678/) (2023)
- [Generative Slate Recommendation with Reinforcement Learning](https://mlsanity.com/paper/2301.08632/) (2023)
- [Causal Decision Transformer for Recommender Systems via Offline Reinforcement Learning](https://mlsanity.com/paper/2304.07920/) (2023)
- [Reinforcement Learning to Optimize Lifetime Value in Cold-Start Recommendation](https://mlsanity.com/paper/2108.09141/) (2021)
- [Beyond Black-Box Advice: Learning-Augmented Algorithms for MDPs with Q-Value Predictions](https://mlsanity.com/paper/2307.10524/) (2023)
- [Generative Adversarial Reward Learning for Generalized Behavior Tendency Inference](https://mlsanity.com/paper/2105.00822/) (2021)
- [Equipping Black-Box Policies with Model-Based Advice for Stable Nonlinear Control](https://mlsanity.com/paper/2206.01341/) (2022)
- [ReRoGCRL: Representation-based Robustness in Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2312.07392/) (2023)
- [RL4RS: A Real-World Dataset for Reinforcement Learning based Recommender System](https://mlsanity.com/paper/2110.11073/) (2021)
- [Synthetic Data-Based Simulators for Recommender Systems: A Survey](https://mlsanity.com/paper/2206.11338/) (2022)
- [Reinforcement Learning with a Disentangled Universal Value Function for Item Recommendation](https://mlsanity.com/paper/2104.02981/) (2021)
- [Navigating Data Corruption in Machine Learning: Balancing Quality, Quantity, and Imputation Strategies](https://mlsanity.com/paper/2412.18296/) (2024)
- [Model-based Offline Policy Optimization with Adversarial Network](https://mlsanity.com/paper/2309.02157/) (2023)
- [Retentive Decision Transformer with Adaptive Masking for Reinforcement Learning based Recommendation Systems](https://mlsanity.com/paper/2403.17634/) (2024)
- [Effective Dimension in Bandit Problems under Censorship](https://mlsanity.com/paper/2302.06916/) (2023)
- [On Minimizing Adversarial Counterfactual Error in Adversarial RL](https://mlsanity.com/paper/2406.04724/) (2024)
- [Conditional Sequential Slate Optimization](https://mlsanity.com/paper/2108.05618/) (2021)
- [Model-free Reinforcement Learning with Stochastic Reward Stabilization for Recommender Systems](https://mlsanity.com/paper/2308.13246/) (2023)
- [A generative recommender system with GMM prior for cancer drug generation and sensitivity prediction](https://mlsanity.com/paper/2206.03555/) (2022)
- [Robust Reinforcement Learning Objectives for Sequential Recommender Systems](https://mlsanity.com/paper/2305.18820/) (2023)
- [Reward Dropout Improves Control: Bi-objective Perspective on Reinforced LM](https://mlsanity.com/paper/2310.04483/) (2023)
- [A General Neural Causal Model for Interactive Recommendation](https://mlsanity.com/paper/2310.19519/) (2023)
- [Adversarial Batch Inverse Reinforcement Learning: Learn to Reward from Imperfect Demonstration for Interactive Recommendation](https://mlsanity.com/paper/2310.19536/) (2023)
- [Aligning GPTRec with Beyond-Accuracy Goals with Reinforcement Learning](https://mlsanity.com/paper/2403.04875/) (2024)

## Abstract (arXiv)

Reinforcement learning is well suited for optimizing policies of recommender systems. Current solutions mostly focus on model-free approaches, which require frequent interactions with the real environment, and thus are expensive in model learning. Offline evaluation methods, such as importance sampling, can alleviate such limitations, but usually request a large amount of logged data and do not work well when the action space is large. In this work, we propose a model-based reinforcement learning solution which models user-agent interaction for offline policy learning via a generative adversarial network. To reduce bias in the learned model and policy, we use a discriminator to evaluate the quality of generated data and scale the generated rewards. Our theoretical analysis and empirical evaluations demonstrate the effectiveness of our solution in learning policies from the offline and generated data.