mlsanityData to 7 Oct 2026

Accelerating Offline Reinforcement Learning Application in Real-Time Bidding and Recommendation: Potential Use of Simulation

arXiv 2109.08331 ↗First posted 17 Sep 2021cs.LGcited by 16

On the map: Offline RL and imitation learning from data → Decision transformers and sequence modeling

Abstract (arXiv)

In recommender systems (RecSys) and real-time bidding (RTB) for online advertisements, we often try to optimize sequential decision making using bandit and reinforcement learning (RL) techniques. In these applications, offline reinforcement learning (offline RL) and off-policy evaluation (OPE) are beneficial because they enable safe policy optimization using only logged data without any risky online interaction. In this position paper, we explore the potential of using simulation to accelerate practical research of offline RL and OPE, particularly in RecSys and RTB. Specifically, we discuss how simulation can help us conduct empirical research of offline RL and OPE. We take a position to argue that we should effectively use simulations in the empirical research of offline RL and OPE. To refute the counterclaim that experiments using only real-world data are preferable, we first point out the underlying risks and reproducibility issue in real-world experiments. Then, we describe how these issues can be addressed by using simulations. Moreover, we show how to incorporate the benefits of both real-world and simulation-based experiments to defend our position. Finally, we also present an open challenge to further facilitate practical research of offline RL and OPE in RecSys and RTB, with respect to public simulation platforms. As a possible solution for the issue, we show our ongoing open source project and its potential use case. We believe that building and utilizing simulation-based evaluation platforms for offline RL and OPE will be of great interest and relevance for the RecSys and RTB community.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data. Builds on 46 of its 55 references that are in the map. First posted on arXiv 17 Sep 2021.

Builds on

46 of 55 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021DeepThermal: Combustion Optimization for Thermal Power Generating Units Using Offline Reinforcement Learning23 Feb

Other references · 45

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  2. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  3. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  5. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  6. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  7. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  8. 2020Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Surveycited by 388
  9. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 365
  10. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
  11. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 285
  12. 2019Batch Policy Learning under Constraintscited by 242
33 more
  1. 2018Top-K Off-Policy Correction for a REINFORCE Recommender Systemcited by 181
  2. 2015Doubly Robust Policy Evaluation and Optimizationcited by 158
  3. 2020Model-Based Offline Planningcited by 126
  4. 2019Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learningcited by 121
  5. 2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
  6. 2020Hyperparameter Selection for Offline Reinforcement Learningcited by 113
  7. 2016Optimal and Adaptive Off-policy Evaluation in Contextual Banditscited by 111
  8. 2019Benchmarking Batch Deep Reinforcement Learning Algorithmscited by 108
  9. 2018Deep Reinforcement Learning for Page-wise Recommendationscited by 99
  10. 2018Offline A/B testing for Recommender Systemscited by 94
  11. 2019RecSim: A Configurable Simulation Platform for Recommender Systemscited by 91
  12. 2017Real-Time Bidding by Reinforcement Learning in Display Advertisingcited by 90
  13. 2019Reinforcement Learning to Optimize Long-term User Engagement in Recommender Systemscited by 90
  14. 2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 90
  15. 2021Representation Matters: Offline Pretraining for Sequential Decision Makingcited by 90
  16. 2021Benchmarks for Deep Off-Policy Evaluationcited by 89
  17. 2019Doubly robust off-policy evaluation with shrinkagecited by 87
  18. 2020Open Bandit Dataset and Pipeline: Towards Realistic and Reproducible Off-Policy Evaluationcited by 77
  19. 2018Recommendations with Negative Feedback via Pairwise Deep Reinforcement Learningcited by 71
  20. 2018Real-Time Bidding with Multi-Agent Reinforcement Learning in Display Advertisingcited by 65
  21. 2018RecoGym: A Reinforcement Learning Environment for the problem of Product Recommendation in Online Advertisingcited by 65
  22. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learningcited by 63
  23. 2018Budget Constrained Bidding by Model-free Reinforcement Learning in Display Advertisingcited by 60
  24. 2019DEAR: Deep Reinforcement Learning for Online Advertising Impression in Recommender Systemscited by 44
  25. 2017Deep Reinforcement Learning for List-wise Recommendationscited by 35
  26. 2020Counterfactual Evaluation of Slate Recommendations with Sequential Reward Interactionscited by 35
  27. 2018Deep Reinforcement Learning for Sponsored Search Real-time Biddingcited by 34
  28. 2021Regularized Behavior Value Estimationcited by 32
  29. 2021Evaluating the Robustness of Off-Policy Evaluationcited by 32
  30. 2018Deep reinforcement learning for search, recommendation, and online advertising: a surveycited by 23
  31. 2020Dynamic Knapsack Optimization Towards Efficient Multi-Channel Sequential Advertisingcited by 16
  32. 2020MARS-Gym: A Gym framework to model, train, and evaluate Recommender Systems for Marketplacescited by 13
  33. 2021Interpretable performance analysis towards offline reinforcement learning: A dataset perspectivecited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 4; 2024: 3; 2025: 3; 2026: 3.

16 papers.

Sort:

By area:

  1. 2023User Behavior Simulation with Large Language Model based Agentscited by 136
  2. 2022Doubly Robust Off-Policy Evaluation for Ranking Policies under the Cascade Behavior Modelcited by 30
  3. 2022Policy-Adaptive Estimator Selection for Off-Policy Evaluationcited by 24
  4. 2025Generative Auto-Bidding with Value-Guided Explorationscited by 23
  5. 2024Off-Policy Evaluation of Slate Bandit Policies via Optimizing Abstractioncited by 17
  6. 2023Off-Policy Evaluation of Ranking Policies under Diverse User Behaviorcited by 16
  7. 2023Towards Assessing and Benchmarking Risk-Return Tradeoff of Off-Policy Evaluationcited by 13
  8. 2023SCOPE-RL: A Python Library for Offline Reinforcement Learning and Off-Policy Evaluationcited by 6
  9. 2022Synthetic Data-Based Simulators for Recommender Systems: A Surveycited by 5
  10. 2024Hyperparameter Optimization Can Even be Harmful in Off-Policy Learning and How to Deal with Itcited by 5
  11. 2024Automated Off-Policy Estimator Selection via Supervised Learningcited by 2
  12. 2025Multi-task Offline Reinforcement Learning for Online Advertising in Recommender Systemscited by 2
4 more
  1. 2026PRO-Bid: Pareto-Prioritized Regret Optimization for Constraint-Aware Generative Auto-Biddingcited by 1
  2. 2025Safely Exploring Novel Actions in Recommender Systems via Deployment-Efficient Policy Learningcited by 0
  3. 2026Constrained Auto-Bidding via Generative Response Modelingcited by 0
  4. 2026Generative Optimization for Incentivized Advertising with Global Level Constraintscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023On the Opportunities and Challenges of Offline Reinforcement Learning for Recommender Systems
  2. 2021Active Offline Policy Selection
  3. 2025Reward Balancing Revisited: Enhancing Offline Reinforcement Learning for Recommender Systems
  4. 2020Offline Reinforcement Learning Hands-On
  5. 2024ROLeR: Effective Reward Shaping in Offline Reinforcement Learning for Recommender Systems
  6. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learning
  7. 2021Showing Your Offline Reinforcement Learning Work: Online Evaluation Budget Matters
  8. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory