mlsanityData to 7 Oct 2026

Dealing with the Unknown: Pessimistic Offline Reinforcement Learning

arXiv 2111.05440 ↗First posted 9 Nov 2021cs.LG cs.ROcited by 18

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Reinforcement Learning (RL) has been shown effective in domains where the agent can learn policies by actively interacting with its operating environment. However, if we change the RL scheme to offline setting where the agent can only update its policy via static datasets, one of the major issues in offline reinforcement learning emerges, i.e. distributional shift. We propose a Pessimistic Offline Reinforcement Learning (PessORL) algorithm to actively lead the agent back to the area where it is familiar by manipulating the value function. We focus on problems caused by out-of-distribution (OOD) states, and deliberately penalize high values at states that are absent in the training dataset, so that the learned pessimistic value function lower bounds the true value anywhere within the state space. We evaluate the PessORL algorithm on various benchmark tasks, where we show that our method gains better performance by explicitly handling OOD states, when compared to those methods merely considering OOD actions.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 31 of its 37 references that are in the map. First posted on arXiv 9 Nov 2021.

Builds on

31 of 37 references.

Closest ancestors · 7

Same direction, newest first

  1. 2020PLAS: Latent Action Space for Offline Reinforcement Learning14 Nov
  2. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  3. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  5. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  6. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb
  7. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 24

  1. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  2. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  4. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  5. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  6. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  7. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  8. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  9. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  10. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  11. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  12. 2016Deep Exploration via Bootstrapped DQNcited by 518
12 more
  1. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 342
  2. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  3. 2015Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Imagescited by 266
  4. 2020Can Autonomous Vehicles Identify, Recover From, and Adapt to Distribution Shifts?cited by 139
  5. 2020BADGR: An Autonomous Self-Supervised Learning-Based Navigation Systemcited by 97
  6. 2017Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learningcited by 86
  7. 2020COG: Connecting New Skills to Past Experience with Offline Reinforcement Learningcited by 83
  8. 2017The Uncertainty Bellman Equation and Explorationcited by 81
  9. 2020Optimistic Exploration even with a Pessimistic Initialisationcited by 25
  10. 2019From Importance Sampling to Doubly Robust Policy Gradientcited by 17
  11. 2019Trajectory-wise Control Variates for Variance Reduction in Policy Gradient Methodscited by 11
  12. 2018Reward-estimation variance elimination in sequential decision processescited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 8; 2024: 4; 2025: 3; 2026: 1.

18 papers.

Sort:

By area:

  1. 2022When to Trust Your Simulator: Dynamics-Aware Hybrid Offline-and-Online Reinforcement Learningcited by 59
  2. 2022Hierarchical Planning Through Goal-Conditioned Offline Reinforcement Learningcited by 48
  3. 2023Diffusion Policies for Out-of-Distribution Generalization in Offline Reinforcement Learningcited by 40
  4. 2024ODICE: Revealing the Mystery of Distribution Correction Estimation via Orthogonal-gradient Updatecited by 21
  5. 2023Guided Online Distillation: Promoting Safe Reinforcement Learning by Offline Demonstrationcited by 14
  6. 2023Residual Q-Learning: Offline and Online Policy Customization without Valuecited by 9
  7. 2023Design from Policies: Conservative Test-Time Adaptation for Offline Policy Optimizationcited by 8
  8. 2024ComaDICE: Offline Cooperative Multi-Agent Reinforcement Learning with Stationary Distribution Shift Regularizationcited by 8
  9. 2024Adaptive Prediction Ensemble: Improving Out-of-Distribution Generalization of Motion Forecastingcited by 7
  10. 2024Uncertainty-Penalized Direct Preference Optimizationcited by 6
  11. 2023DOMAIN: MilDly COnservative Model-BAsed OfflINe Reinforcement Learningcited by 4
  12. 2025An Optimal Discriminator Weighted Imitation Perspective for Reinforcement Learningcited by 4
6 more
  1. 2023Prioritized Trajectory Replay: A Replay Memory for Data-driven Reinforcement Learningcited by 3
  2. 2023An Offline Learning Approach to Propagator Modelscited by 2
  3. 2023SeRO: Self-Supervised Reinforcement Learning for Recovery from Out-of-Distribution Situationscited by 2
  4. 2025LaMOuR: Leveraging Language Models for Out-of-Distribution Recovery in Reinforcement Learningcited by 1
  5. 2025CAWR: Corruption-Averse Advantage-Weighted Regression for Robust Policy Optimizationcited by 0
  6. 2026Reward Learning through Ranking Mean Squared Errorcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2020Is Pessimism Provably Efficient for Offline RL?
  2. 2025Semi-pessimistic Reinforcement Learning
  3. 2022State-Aware Proximal Pessimistic Algorithms for Offline Reinforcement Learning
  4. 2026Pessimistic Auxiliary Policy for Offline Reinforcement Learning
  5. 2020POPO: Pessimistic Offline Policy Optimization
  6. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning
  7. 2023Double Pessimism is Provably Efficient for Distributionally Robust Offline Reinforcement Learning: Generic Algorithm and Robust Partial Coverage
  8. 2024The Virtues of Pessimism in Inverse Reinforcement Learning