mlsanityData to 7 Oct 2026

Percentile Criterion Optimization in Offline Reinforcement Learning

arXiv 2404.05055 ↗First posted 7 Apr 2024cs.LG cs.AIcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

In reinforcement learning, robust policies for high-stakes decision-making problems with limited data are usually computed by optimizing the percentile criterion. The percentile criterion is approximately solved by constructing an ambiguity set that contains the true model with high probability and optimizing the policy for the worst model in the set. Since the percentile criterion is non-convex, constructing ambiguity sets is often challenging. Existing work uses Bayesian credible regions as ambiguity sets, but they are often unnecessarily large and result in learning overly conservative policies. To overcome these shortcomings, we propose a novel Value-at-Risk based dynamic programming algorithm to optimize the percentile criterion without explicitly constructing any ambiguity sets. Our theoretical and empirical results show that our algorithm implicitly constructs much smaller ambiguity sets and learns less conservative robust policies.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 14 of its 51 references that are in the map. First posted on arXiv 7 Apr 2024.

Builds on

14 of 51 references.

References · 14

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2021Convex optimizationcited by 2,479
  4. 2014Algorithms for CVaR Optimization in MDPscited by 98
  5. 2019Robust Reinforcement Learning for Continuous Control with Model Misspecificationcited by 76
  6. 2015Policy Gradient for Coherent Risk Measurescited by 74
  7. 2016Safe Policy Improvement by Minimizing Robust Baseline Regretcited by 51
  8. 2018Soft-Robust Actor-Critic Policy-Gradientcited by 37
  9. 2019Beyond Confidence Regions: Tight Bayesian Ambiguity Sets for Robust MDPscited by 37
  10. 2019A Bayesian Approach to Robust Reinforcement Learningcited by 32
  11. 2018Robust Markov Decision Process: Beyond Rectangularitycited by 24
  12. 2014Policy Gradients for CVaR-Constrained MDPscited by 20
2 more
  1. 2020First-Order Methods for Wasserstein Distributionally Robust MDPcited by 16
  2. 2022A policy gradient approach for optimization of smooth risk measurescited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Selective Uncertainty Propagation in Offline RL
  2. 2023Achieving the Asymptotically Optimal Sample Complexity of Offline Reinforcement Learning: A DRO-Based Approach
  3. 2026Bayesian Conservative Policy Optimization (BCPO): A Novel Uncertainty-Calibrated Offline Reinforcement Learning with Credible Lower Bounds
  4. 2024Pessimism Meets Risk: Risk-Sensitive Offline Reinforcement Learning
  5. 2022On Practical Reinforcement Learning: Provable Robustness, Scalability, and Statistical Efficiency
  6. 2024Minimax Optimal and Computationally Efficient Algorithms for Distributionally Robust Offline Reinforcement Learning
  7. 2024Diverse Randomized Value Functions: A Provably Pessimistic Approach for Offline Reinforcement Learning
  8. 2022Optimal Conservative Offline RL with General Function Approximation via Augmented Lagrangian