mlsanityData to 14 Oct 2026

Active Offline Policy Selection

arXiv 2106.10251 ↗First posted 18 Jun 2021cs.LG cs.AI stat.MLcited by 26

Not on the map. Why?

Abstract (arXiv)

This paper addresses the problem of policy selection in domains with abundant logged data, but with a restricted interaction budget. Solving this problem would enable safe evaluation and deployment of offline reinforcement learning policies in industry, robotics, and recommendation domains among others. Several off-policy evaluation (OPE) techniques have been proposed to assess the value of policies using only logged data. However, there is still a big gap between the evaluation by OPE and the full online evaluation. Yet, large amounts of online interactions are often not possible in practice. To overcome this problem, we introduce active offline policy selection - a novel sequential decision approach that combines logged data with online interaction to identify the best policy. We use OPE estimates to warm start the online evaluation. Then, in order to utilize the limited environment interactions wisely we decide which policy to evaluate next based on a Bayesian optimization method with a kernel that represents policy similarity. We use multiple benchmarks, including real-world robotics, with a large number of candidate policies to show that the proposed approach improves upon state-of-the-art OPE estimates and pure online policy evaluation.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Builds on 41 of its 73 references that are in the map. First posted on arXiv 18 Jun 2021.

Builds on

41 of 73 references.

References · 41

  1. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  3. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  4. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  5. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  6. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  7. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  8. 2015End-to-End Training of Deep Visuomotor Policiescited by 875
  9. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
  10. 2020MOPO: Model-based Offline Policy Optimizationcited by 636
  11. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 590
  12. 2020dm_control: Software and Tasks for Continuous Controlcited by 375
29 more
  1. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 369
  2. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 339
  3. 2019Challenges of Real-World Reinforcement Learningcited by 291
  4. 2020Critic Regularized Regressioncited by 275
  5. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 264
  6. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  7. 2019Batch Policy Learning under Constraintscited by 244
  8. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 227
  9. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learningcited by 215
  10. 2018Distributed Distributional Deterministic Policy Gradientscited by 199
  11. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  12. 2020Acme: A Research Framework for Distributed Reinforcement Learningcited by 146
  13. 2018More Robust Doubly Robust Off-policy Evaluationcited by 141
  14. 2019Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learningcited by 121
  15. 2020Effective Diversity in Population Based Reinforcement Learningcited by 119
  16. 2020Hyperparameter Selection for Offline Reinforcement Learningcited by 114
  17. 2015An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learningcited by 103
  18. 2021Benchmarks for Deep Off-Policy Evaluationcited by 89
  19. 2020Off-Policy Evaluation via the Regularized Lagrangiancited by 86
  20. 2018Bayesian Optimization in AlphaGocited by 55
  21. 2019Learning When-to-Treat Policiescited by 49
  22. 2020Ready Policy One: World Building Through Active Learningcited by 33
  23. 2021Autoregressive Dynamics Models for Offline Policy Evaluation and Optimizationcited by 33
  24. 2021Regularized Behavior Value Estimationcited by 32
  25. 2020Offline Policy Selection under Uncertaintycited by 31
  26. 2020Policy Evaluation Networkscited by 28
  27. 2021Reverb: A Framework For Experience Replaycited by 27
  28. 2020Active Reinforcement Learning: Observing Rewards at a Costcited by 20
  29. 2018Discovering General-Purpose Active Learning Strategiescited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 5; 2023: 5; 2024: 4; 2025: 5; 2026: 6.

26 papers.

Sort:

By area:

  1. 2022Supported Policy Optimization for Offline Reinforcement Learningcited by 74
  2. 2022An Empirical Study of Implicit Regularization in Deep Offline RLcited by 24
  3. 2022Real World Offline Reinforcement Learning with Realistic Data Sourcecited by 17
  4. 2023Blending Imitation and Reinforcement Learning for Robust Policy Improvementcited by 17
  5. 2021Robust On-Policy Sampling for Data-Efficient Policy Evaluation in Reinforcement Learningcited by 15
  6. 2023Efficient Policy Evaluation with Offline Data Informed Behavior Policy Designcited by 14
  7. 2022User-Interactive Offline Reinforcement Learningcited by 11
  8. 2023Active Policy Improvement from Multiple Black-box Oraclescited by 11
  9. 2025A Clean Slate for Offline Reinforcement Learningcited by 11
  10. 2022Comparing Model-free and Model-based Algorithms for Offline Reinforcement Learningcited by 9
  11. 2023On-Policy Policy Gradient Reinforcement Learning Without On-Policy Samplingcited by 6
  12. 2024Logarithmic Neyman Regret for Adaptive Estimation of the Average Treatment Effectcited by 4
14 more
  1. 2024Kernel Metric Learning for In-Sample Off-Policy Evaluation of Deterministic RL Policiescited by 3
  2. 2024Predictive Performance Comparison of Decision Policies Under Confoundingcited by 2
  3. 2024Active Reinforcement Learning Strategies for Offline Policy Improvementcited by 2
  4. 2025Optimistic Algorithms for Adaptive Estimation of the Average Treatment Effectcited by 2
  5. 2026Predicting Closed-Loop Performance of Latent World Models: Offline Checkpoint Selection for MPC and Model-Based RL Under Non-Markovian Rewards in LunarLandercited by 2
  6. 2023π2vec: Policy Representations with Successor Featurescited by 1
  7. 2025Enhancing Offline Model-Based RL via Active Model Selection: A Bayesian Optimization Perspectivecited by 1
  8. 2026Adaptive Policy Selection and Fine-Tuning under Interaction Budgets for Offline-to-Online Reinforcement Learningcited by 1
  9. 2026A More Accurate Algorithm Comparison through A/B Testing using Offline Evaluation Methodscited by 1
  10. 2025Centralized Adaptive Sampling for Reliable Co-Training of Independent Multi-Agent Policiescited by 0
  11. 2025Which Rewards Matter? Reward Selection for Reinforcement Learning under Limited Feedbackcited by 0
  12. 2026Regression Test Selection for Updated Capability Modules in Compositional ML Systems via Atomic-Quality Probescited by 0
  13. 2026Active Offline-to-Online Reinforcement Learningcited by 0
  14. 2026StepCOPS: Closed-Testing Lower-Tail Certificates for Language-Model Policy Selectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024OPERA: Automatic Offline Policy Evaluation with Re-weighted Aggregates of Multiple Estimators
  2. 2025Model Selection for Off-policy Evaluation: New Algorithms and Experimental Protocol
  3. 2026Formulating Reinforcement Learning for Human-Robot Collaboration through Off-Policy Evaluation
  4. 2022Bridging the Gap Between Offline and Online Reinforcement Learning Evaluation Methodologies
  5. 2021Accelerating Offline Reinforcement Learning Application in Real-Time Bidding and Recommendation: Potential Use of Simulation
  6. 2023Boosting Offline Reinforcement Learning with Action Preference Query
  7. 2023Offline Policy Evaluation for Reinforcement Learning with Adaptively Collected Data
  8. 2021Measuring Data Quality for Dataset Selection in Offline Reinforcement Learning