mlsanityData to 7 Oct 2026

Pessimistic Model Selection for Offline Deep Reinforcement Learning

arXiv 2111.14346 ↗First posted 29 Nov 2021cs.LG cs.AI cs.CEcited by 3

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Deep Reinforcement Learning (DRL) has demonstrated great potentials in solving sequential decision making problems in many applications. Despite its promising performance, practical gaps exist when deploying DRL in real-world scenarios. One main barrier is the over-fitting issue that leads to poor generalizability of the policy learned by DRL. In particular, for offline DRL with observational data, model selection is a challenging task as there is no ground truth available for performance demonstration, in contrast with the online setting with simulated environments. In this work, we propose a pessimistic model selection (PMS) approach for offline DRL with a theoretical guarantee, which features a provably effective framework for finding the best policy among a set of candidate models. Two refined approaches are also proposed to address the potential bias of DRL model in identifying the optimal policy. Numerical studies demonstrated the superior performance of our approach over existing methods.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 47 of its 71 references that are in the map. First posted on arXiv 29 Nov 2021.

Builds on

47 of 71 references.

Closest ancestors · 15

Same direction, newest first

  1. 2021Proximal Reinforcement Learning: Efficient Off-Policy Evaluation in Partially Observed Markov Decision Processes28 Oct
  2. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning19 Aug
  3. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
  4. 2021Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiency5 Feb
  5. 2021Fast Rates for the Regret of Offline Reinforcement Learning31 Jan
  6. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  7. 2020Batch Value-function Approximation with Only Realizability11 Aug
  8. 2020Hyperparameter Selection for Offline Reinforcement Learning17 Jul
  9. 2020RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learning24 Jun
  10. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  11. 2020MOPO: Model-based Offline Policy Optimization27 May
  12. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
3 more
  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  2. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb
  3. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 32

  1. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  2. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  4. 2015Prioritized Experience Replaycited by 1,070
  5. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  6. 2017Deep Reinforcement Learning that Matterscited by 727
  7. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  8. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
  9. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  10. 2019Batch Policy Learning under Constraintscited by 242
  11. 2018Unity: A General Platform for Intelligent Agentscited by 232
  12. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 226
20 more
  1. 2019A Theoretical Analysis of Deep Q-Learningcited by 212
  2. 2018Distributed Distributional Deterministic Policy Gradientscited by 195
  3. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  4. 2019Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learningcited by 121
  5. 2020Human-centric Dialog Training via Offline Reinforcement Learningcited by 92
  6. 2017Self-supervised Deep Reinforcement Learning with Generalized Computation Graphs for Robot Navigationcited by 74
  7. 2020Batch Policy Learning in Average Reward Markov Decision Processescited by 69
  8. 2021Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settingscited by 68
  9. 2020Statistical Inference of the Value Function for Reinforcement Learning in Infinite Horizon Settingscited by 65
  10. 2018Evaluating Reinforcement Learning Algorithms in Observational Health Settingscited by 60
  11. 2019Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimationcited by 48
  12. 2020Adaptive Estimator Selection for Off-Policy Evaluationcited by 38
  13. 2020Confident Off-Policy Evaluation and Selection through Self-Normalized Importance Weightingcited by 38
  14. 2020Offline Policy Selection under Uncertaintycited by 31
  15. 2020Online Model Selection for Reinforcement Learning with Function Approximationcited by 28
  16. 2020Clinician-in-the-Loop Decision Making: Reinforcement Learning with Near-Optimal Set-Valued Policiescited by 24
  17. 2021A Survey of Deep Reinforcement Learning in Recommender Systems: A Systematic Review and Future Directionscited by 19
  18. 2020Ultrasound-Guided Robotic Navigation with Deep Reinforcement Learningcited by 16
  19. 2020Enhanced Adversarial Strategically-Timed Attacks against Deep Reinforcement Learningcited by 13
  20. 2014Exact Post Model Selection Inference for Marginal Screeningcited by 12

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 1.

3 papers.

  1. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learningcited by 103
  2. 2022Policy-Adaptive Estimator Selection for Off-Policy Evaluationcited by 24
  3. 2022Bellman Residual Orthogonalization for Offline Reinforcement Learningcited by 10

Similar, not cited

Close in wording, no citation either way

  1. 2023Neural Network Approximation for Pessimistic Offline Reinforcement Learning
  2. 2023Model-based Offline Reinforcement Learning with Local Misspecification
  3. 2022Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Belief
  4. 2022Oracle Inequalities for Model Selection in Offline Reinforcement Learning
  5. 2021COMBO: Conservative Offline Model-Based Policy Optimization
  6. 2023Double Pessimism is Provably Efficient for Distributionally Robust Offline Reinforcement Learning: Generic Algorithm and Robust Partial Coverage
  7. 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learning
  8. 2024Optimistic Model Rollouts for Pessimistic Offline Policy Optimization