mlsanityData to 14 Oct 2026

Optimistic Policy Regularization

arXiv 2603.06793 ↗First posted 6 Mar 2026cs.LG cs.AIcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Deep reinforcement learning agents frequently suffer from premature convergence, where early entropy collapse causes the policy to discard exploratory behaviors before discovering globally optimal strategies. We introduce Optimistic Policy Regularization (OPR), a lightweight mechanism designed to preserve and reinforce historically successful trajectories during policy optimization. OPR maintains a dynamic buffer of high-performing episodes and biases learning toward these behaviors through directional log-ratio reward shaping and an auxiliary behavioral cloning objective. When instantiated on Proximal Policy Optimization (PPO), OPR substantially improves sample efficiency on the Arcade Learning Environment. Across 49 Atari games evaluated at the 10-million step benchmark, OPR achieves the highest score in 22 environments despite baseline methods being reported at the standard 50-million step horizon. Beyond arcade benchmarks, OPR also generalizes to the CAGE Challenge 2 cyber-defense environment, surpassing the competition-winning Cardiff agent while using the same PPO architecture. These results demonstrate that anchoring policy updates to empirically successful trajectories can improve both sample efficiency and final performance.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 12 of its 16 references that are in the map. First posted on arXiv 6 Mar 2026.

Builds on

12 of 16 references.

References · 12

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015Trust Region Policy Optimizationcited by 2,580
  4. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  5. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  6. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  7. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  8. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  9. 2018Exploration by Random Network Distillationcited by 993
  10. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  11. 2016Sample Efficient Actor-Critic with Experience Replaycited by 188
  12. 2018Self-Imitation Learningcited by 138

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Revisiting Regularized Policy Optimization for Stable and Efficient Reinforcement Learning in Two-Player Games
  2. 2025EPO: Entropy-regularized Policy Optimization for LLM Agents Reinforcement Learning
  3. 2026Robust Regularized Policy Iteration under Transition Uncertainty
  4. 2024Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning
  5. 2025Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RL
  6. 2024The Definitive Guide to Policy Gradients in Deep Reinforcement Learning: Theory, Algorithms and Implementations
  7. 2026FERPO: Forward Entropy-Regularized Policy Optimization
  8. 2024Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous control