# Generative Adversarial User Model for Reinforcement Learning Based Recommendation System

arXiv: https://arxiv.org/abs/1812.10613 · HTML: https://mlsanity.com/paper/1812.10613/

## Key facts

- Cited by 64 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Dec 2018.

## Built on by (most cited first, 30 of 64)

- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble](https://mlsanity.com/paper/2110.01548/) (2021)
- [User Behavior Simulation with Large Language Model based Agents](https://mlsanity.com/paper/2306.02552/) (2023)
- [Advances and Challenges in Conversational Recommender Systems: A Survey](https://mlsanity.com/paper/2101.09459/) (2021)
- [On Generative Agents in Recommendation](https://mlsanity.com/paper/2310.10108/) (2023)
- [Elastic Decision Transformer](https://mlsanity.com/paper/2307.02484/) (2023)
- [A Review of Modern Recommender Systems Using Generative Models (Gen-RecSys)](https://mlsanity.com/paper/2404.00579/) (2024)
- [RS-Agent: Automating Remote Sensing Tasks through Intelligent Agent](https://mlsanity.com/paper/2406.07089/) (2024)
- [On Targeted Manipulation and Deception when Optimizing LLMs for User Feedback](https://mlsanity.com/paper/2411.02306/) (2024)
- [Recent Advances in Generative AI and Large Language Models: Current Status, Challenges, and Perspectives](https://mlsanity.com/paper/2407.14962/) (2024)
- [Estimating and Penalizing Induced Preference Shifts in Recommender Systems](https://mlsanity.com/paper/2204.11966/) (2022)
- [Prototypical Reward Network for Data-Efficient RLHF](https://mlsanity.com/paper/2406.06606/) (2024)
- [A Survey of Deep Reinforcement Learning in Recommender Systems: A Systematic Review and Future Directions](https://mlsanity.com/paper/2109.03540/) (2021)
- [A General Offline Reinforcement Learning Framework for Interactive Recommendation](https://mlsanity.com/paper/2310.00678/) (2023)
- [Rethinking ValueDice: Does It Really Improve Performance?](https://mlsanity.com/paper/2202.02468/) (2022)
- [Choosing the Best of Both Worlds: Diverse and Novel Recommendations through Multi-Objective Reinforcement Learning](https://mlsanity.com/paper/2110.15097/) (2021)
- [Generative Slate Recommendation with Reinforcement Learning](https://mlsanity.com/paper/2301.08632/) (2023)
- [Adversarial Counterfactual Environment Model Learning](https://mlsanity.com/paper/2206.04890/) (2022)
- [AI Agent for Education: von Neumann Multi-Agent System Framework](https://mlsanity.com/paper/2501.00083/) (2024)
- [SUBER: An RL Environment with Simulated Human Behavior for Recommender Systems](https://mlsanity.com/paper/2406.01631/) (2024)
- [A Versatile Multi-Agent Reinforcement Learning Benchmark for Inventory Management](https://mlsanity.com/paper/2306.07542/) (2023)
- [Supervised Advantage Actor-Critic for Recommender Systems](https://mlsanity.com/paper/2111.03474/) (2021)
- [Optimizing Audio Recommendations for the Long-Term: A Reinforcement Learning Perspective](https://mlsanity.com/paper/2302.03561/) (2023)
- [How To Guide Your Learner: Imitation Learning with Active Adaptive Expert Involvement](https://mlsanity.com/paper/2303.02073/) (2023)
- [Generative Adversarial Reward Learning for Generalized Behavior Tendency Inference](https://mlsanity.com/paper/2105.00822/) (2021)
- [Reinforcement Learning in Practice: Opportunities and Challenges](https://mlsanity.com/paper/2202.11296/) (2022)
- [Meta Policy Learning for Cold-Start Conversational Recommendation](https://mlsanity.com/paper/2205.11788/) (2022)
- [Optimizing Long-term Value for Auction-Based Recommender Systems via On-Policy Reinforcement Learning](https://mlsanity.com/paper/2305.13747/) (2023)
- [Provably and Practically Efficient Adversarial Imitation Learning with General Function Approximation](https://mlsanity.com/paper/2411.00610/) (2024)
- [RL4RS: A Real-World Dataset for Reinforcement Learning based Recommender System](https://mlsanity.com/paper/2110.11073/) (2021)

## Abstract (arXiv)

There are great interests as well as many challenges in applying reinforcement learning (RL) to recommendation systems. In this setting, an online user is the environment; neither the reward function nor the environment dynamics are clearly defined, making the application of RL challenging. In this paper, we propose a novel model-based reinforcement learning framework for recommendation systems, where we develop a generative adversarial network to imitate user behavior dynamics and learn her reward function. Using this user model as the simulation environment, we develop a novel Cascading DQN algorithm to obtain a combinatorial recommendation policy which can handle a large number of candidate items efficiently. In our experiments with real data, we show this generative adversarial user model can better explain user behavior than alternatives, and the RL policy based on this model can lead to a better long-term reward for the user and higher click rate for the system.