# RecoGym: A Reinforcement Learning Environment for the problem of Product Recommendation in Online Advertising

arXiv: https://arxiv.org/abs/1808.00720 · HTML: https://mlsanity.com/paper/1808.00720/

## Key facts

- Cited by 65 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Aug 2018.

## Built on by (most cited first, 30 of 65)

- [User Behavior Simulation with Large Language Model based Agents](https://mlsanity.com/paper/2306.02552/) (2023)
- [On Generative Agents in Recommendation](https://mlsanity.com/paper/2310.10108/) (2023)
- [Leveraging Large Language Models in Conversational Recommender Systems](https://mlsanity.com/paper/2305.07961/) (2023)
- [LLM-Powered User Simulator for Recommender System](https://mlsanity.com/paper/2412.16984/) (2024)
- [A Survey of Sim-to-Real Methods in RL: Progress, Prospects and Challenges with Foundation Models](https://mlsanity.com/paper/2502.13187/) (2025)
- [Fairness in Recommender Systems: Research Landscape and Future Directions](https://mlsanity.com/paper/2205.11127/) (2022)
- [What are you optimizing for? Aligning Recommender Systems with Human Values](https://mlsanity.com/paper/2107.10939/) (2021)
- [RecSim NG: Toward Principled Uncertainty Modeling for Recommender Ecosystems](https://mlsanity.com/paper/2103.08057/) (2021)
- [A Survey of Deep Reinforcement Learning in Recommender Systems: A Systematic Review and Future Directions](https://mlsanity.com/paper/2109.03540/) (2021)
- [Fair ranking: a critical review, challenges, and future directions](https://mlsanity.com/paper/2201.12662/) (2022)
- [A Survey on LLM-based Conversational User Simulation](https://mlsanity.com/paper/2604.24977/) (2026)
- [Accelerating Offline Reinforcement Learning Application in Real-Time Bidding and Recommendation: Potential Use of Simulation](https://mlsanity.com/paper/2109.08331/) (2021)
- [Deep Exploration for Recommendation Systems](https://mlsanity.com/paper/2109.12509/) (2021)
- [On (Normalised) Discounted Cumulative Gain as an Off-Policy Evaluation Metric for Top-n Recommendation](https://mlsanity.com/paper/2307.15053/) (2023)
- [Optimal Baseline Corrections for Off-Policy Contextual Bandits](https://mlsanity.com/paper/2405.05736/) (2024)
- [T-RECS: A Simulation Tool to Study the Societal Impact of Recommender Systems](https://mlsanity.com/paper/2107.08959/) (2021)
- [No-Regret Algorithms for Private Gaussian Process Bandit Optimization](https://mlsanity.com/paper/2102.12467/) (2021)
- [Accounting for AI and Users Shaping One Another: The Role of Mathematical Models](https://mlsanity.com/paper/2404.12366/) (2024)
- [SUBER: An RL Environment with Simulated Human Behavior for Recommender Systems](https://mlsanity.com/paper/2406.01631/) (2024)
- [A Versatile Multi-Agent Reinforcement Learning Benchmark for Inventory Management](https://mlsanity.com/paper/2306.07542/) (2023)
- [EasyRL4Rec: An Easy-to-use Library for Reinforcement Learning Based Recommender Systems](https://mlsanity.com/paper/2402.15164/) (2024)
- [Supervised Advantage Actor-Critic for Recommender Systems](https://mlsanity.com/paper/2111.03474/) (2021)
- [A preprocessing perspective for quantum machine learning classification advantage using NISQ algorithms](https://mlsanity.com/paper/2208.13251/) (2022)
- [Simulations in Recommender Systems: An industry perspective](https://mlsanity.com/paper/2109.06723/) (2021)
- [Should I send this notification? Optimizing push notifications decision making by modeling the future](https://mlsanity.com/paper/2202.08812/) (2022)
- [Balancing Consumer and Business Value of Recommender Systems: A Simulation-based Analysis](https://mlsanity.com/paper/2203.05952/) (2022)
- [Quantum Computing Provides Exponential Regret Improvement in Episodic Reinforcement Learning](https://mlsanity.com/paper/2302.08617/) (2023)
- [SCOPE-RL: A Python Library for Offline Reinforcement Learning and Off-Policy Evaluation](https://mlsanity.com/paper/2311.18206/) (2023)
- [Towards Personalized Conversational Sales Agents: Contextual User Profiling for Strategic Action](https://mlsanity.com/paper/2504.08754/) (2025)
- [RL4RS: A Real-World Dataset for Reinforcement Learning based Recommender System](https://mlsanity.com/paper/2110.11073/) (2021)

## Abstract (arXiv)

Recommender Systems are becoming ubiquitous in many settings and take many forms, from product recommendation in e-commerce stores, to query suggestions in search engines, to friend recommendation in social networks. Current research directions which are largely based upon supervised learning from historical data appear to be showing diminishing returns with a lot of practitioners report a discrepancy between improvements in offline metrics for supervised learning and the online performance of the newly proposed models. One possible reason is that we are using the wrong paradigm: when looking at the long-term cycle of collecting historical performance data, creating a new version of the recommendation model, A/B testing it and then rolling it out. We see that there a lot of commonalities with the reinforcement learning (RL) setup, where the agent observes the environment and acts upon it in order to change its state towards better states (states with higher rewards). To this end we introduce RecoGym, an RL environment for recommendation, which is defined by a model of user traffic patterns on e-commerce and the users response to recommendations on the publisher websites. We believe that this is an important step forward for the field of recommendation systems research, that could open up an avenue of collaboration between the recommender systems and reinforcement learning communities and lead to better alignment between offline and online performance metrics.