# Toward Simulating Environments in Reinforcement Learning Based Recommendations

arXiv: https://arxiv.org/abs/1906.11462 · HTML: https://mlsanity.com/paper/1906.11462/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Jun 2019.

## Built on by (most cited first, 6 of 6)

- [Advances and Challenges in Conversational Recommender Systems: A Survey](https://mlsanity.com/paper/2101.09459/) (2021)
- [AutoLoss: Automated Loss Function Search in Recommendations](https://mlsanity.com/paper/2106.06713/) (2021)
- [RecSim NG: Toward Principled Uncertainty Modeling for Recommender Ecosystems](https://mlsanity.com/paper/2103.08057/) (2021)
- [A Survey of Deep Reinforcement Learning in Recommender Systems: A Systematic Review and Future Directions](https://mlsanity.com/paper/2109.03540/) (2021)
- [FEBR: Expert-Based Recommendation Framework for beneficial and personalized content](https://mlsanity.com/paper/2108.01455/) (2021)
- [Debiased Model-based Interactive Recommendation](https://mlsanity.com/paper/2402.15819/) (2024)

## Abstract (arXiv)

With the recent advances in Reinforcement Learning (RL), there have been tremendous interests in employing RL for recommender systems. However, directly training and evaluating a new RL-based recommendation algorithm needs to collect users' real-time feedback in the real system, which is time and efforts consuming and could negatively impact on users' experiences. Thus, it calls for a user simulator that can mimic real users' behaviors where we can pre-train and evaluate new recommendation algorithms. Simulating users' behaviors in a dynamic system faces immense challenges -- (i) the underlining item distribution is complex, and (ii) historical logs for each user are limited. In this paper, we develop a user simulator base on Generative Adversarial Network (GAN). To be specific, the generator captures the underlining distribution of users' historical logs and generates realistic logs that can be considered as augmentations of real logs; while the discriminator not only distinguishes real and fake logs but also predicts users' behaviors. The experimental results based on real-world e-commerce data demonstrate the effectiveness of the proposed simulator.