mlsanityData to 7 Oct 2026

POPO: Pessimistic Offline Policy Optimization

arXiv 2012.13682 ↗First posted 26 Dec 2020cs.LG cs.AIcited by 5

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL), also known as batch RL, aims to optimize policy from a large pre-recorded dataset without interaction with the environment. This setting offers the promise of utilizing diverse, pre-collected datasets to obtain policies without costly, risky, active exploration. However, commonly used off-policy algorithms based on Q-learning or actor-critic perform poorly when learning from a static dataset. In this work, we study why off-policy RL methods fail to learn in offline setting from the value function view, and we propose a novel offline RL algorithm that we call Pessimistic Offline Policy Optimization (POPO), which learns a pessimistic value function to get a strong policy. We find that POPO performs surprisingly well and scales to tasks with high-dimensional state and action space, comparing or outperforming several state-of-the-art offline RL algorithms on benchmark tasks.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 27 of its 47 references that are in the map. First posted on arXiv 26 Dec 2020.

Builds on

27 of 47 references.

Closest ancestors · 6

Same direction, newest first

  1. 2020Critic Regularized Regression26 Jun
  2. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  3. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  4. 2019Behavior Regularized Offline Reinforcement Learning26 Nov
  5. 2019Benchmarking Batch Deep Reinforcement Learning Algorithms3 Oct
  6. 2019An Optimistic Perspective on Offline Reinforcement Learning10 Jul

Other references · 21

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  5. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  6. 2016OpenAI Gymcited by 1,684
  7. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  8. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  9. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  10. 2017Deep Reinforcement Learning that Matterscited by 727
  11. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
  12. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
9 more
  1. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
  2. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  3. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
  4. 2020Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPOcited by 184
  5. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 141
  6. 2018Reinforcement Learning from Imperfect Demonstrationscited by 74
  7. 2016Model-Free Imitation Learning with Policy Optimizationcited by 52
  8. 2015Learning the Preferences of Ignorant, Inconsistent Agentscited by 45
  9. 2018Truncated Horizon Policy Search: Combining Reinforcement Learning & Imitation Learningcited by 39

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 0; 2023: 2; 2024: 1.

5 papers.

  1. 2021Online and Offline Reinforcement Learning by Planning with a Learned Modelcited by 90
  2. 2023Adaptive Policy Learning for Offline-to-Online Reinforcement Learningcited by 40
  3. 2021MEPG: A Minimalist Ensemble Policy Gradient Framework for Deep Reinforcement Learningcited by 5
  4. 2023Centralized Cooperative Exploration Policy for Continuous Control Taskscited by 1
  5. 2024A Robust Quantile Huber Loss With Interpretable Parameter Adjustment In Distributional Reinforcement Learningcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2021Dealing with the Unknown: Pessimistic Offline Reinforcement Learning
  2. 2020MOPO: Model-based Offline Policy Optimization
  3. 2020MOReL : Model-Based Offline Reinforcement Learning
  4. 2021A Minimalist Approach to Offline Reinforcement Learning
  5. 2020Is Pessimism Provably Efficient for Offline RL?
  6. 2023Projected Off-Policy Q-Learning (POP-QL) for Stabilizing Offline Reinforcement Learning
  7. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism
  8. 2025Semi-pessimistic Reinforcement Learning