mlsanityData to 7 Oct 2026

Pessimistic Auxiliary Policy for Offline Reinforcement Learning

arXiv 2602.23974 ↗First posted 27 Feb 2026cs.AIcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning aims to learn an agent from pre-collected datasets, avoiding unsafe and inefficient real-time interaction. However, inevitable access to out-ofdistribution actions during the learning process introduces approximation errors, causing the error accumulation and considerable overestimation. In this paper, we construct a new pessimistic auxiliary policy for sampling reliable actions. Specifically, we develop a pessimistic auxiliary strategy by maximizing the lower confidence bound of the Q-function. The pessimistic auxiliary strategy exhibits a relatively high value and low uncertainty in the vicinity of the learned policy, avoiding the learned policy sampling high-value actions with potentially high errors during the learning process. Less approximation error introduced by sampled action from pessimistic auxiliary strategy leads to the alleviation of error accumulation. Extensive experiments on offline reinforcement learning benchmarks reveal that utilizing the pessimistic auxiliary strategy can effectively improve the efficacy of other offline RL approaches.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 29 of its 44 references that are in the map. First posted on arXiv 27 Feb 2026.

Builds on

29 of 44 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025NeoRL-2: Near Real-World Benchmarks for Offline Reinforcement Learning with Extended Realistic Scenarios25 Mar
  2. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  3. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
  4. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  5. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning9 Jun
  6. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning17 May
  7. 2020Critic Regularized Regression26 Jun
  8. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  9. 2020MOPO: Model-based Offline Policy Optimization27 May
  10. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  11. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  12. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
1 more
  1. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 16

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  3. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  5. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  6. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  7. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  8. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  9. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
  10. 2016Deep Exploration via Bootstrapped DQNcited by 518
  11. 2018Is Q-learning Provably Efficient?cited by 485
  12. 2020Maxmin Q-learning: Controlling the Estimation Bias of Q-learningcited by 117
4 more
  1. 2022Bootstrapped Transformer for Offline Reinforcement Learningcited by 37
  2. 2023HarmonyDream: Task Harmonization Inside World Modelscited by 22
  3. 2024Learning Latent Dynamic Robust Representations for World Modelscited by 20
  4. 2020Continuous Action Reinforcement Learning from a Mixture of Interpretable Expertscited by 17

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2021Dealing with the Unknown: Pessimistic Offline Reinforcement Learning
  2. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning
  3. 2022Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexity
  4. 2020Is Pessimism Provably Efficient for Offline RL?
  5. 2022Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Belief
  6. 2025Semi-pessimistic Reinforcement Learning
  7. 2023Double Pessimism is Provably Efficient for Distributionally Robust Offline Reinforcement Learning: Generic Algorithm and Robust Partial Coverage
  8. 2023Pessimistic Nonlinear Least-Squares Value Iteration for Offline Reinforcement Learning