mlsanityData to 7 Oct 2026

Behavior Proximal Policy Optimization

arXiv 2302.11312 ↗First posted 22 Feb 2023cs.LGcited by 43

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) is a challenging setting where existing off-policy actor-critic methods perform poorly due to the overestimation of out-of-distribution state-action pairs. Thus, various additional augmentations are proposed to keep the learned policy close to the offline dataset (or the behavior policy). In this work, starting from the analysis of offline monotonic policy improvement, we get a surprising finding that some online on-policy algorithms are naturally able to solve offline RL. Specifically, the inherent conservatism of these on-policy algorithms is exactly what the offline RL method needs to overcome the overestimation. Based on this, we propose Behavior Proximal Policy Optimization (BPPO), which solves offline RL without any extra constraint or regularization introduced compared to PPO. Extensive experiments on the D4RL benchmark indicate this extremely succinct method outperforms state-of-the-art offline RL algorithms. Our implementation is available at https://github.com/Dragon-Zhuang/BPPO.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 39 of its 50 references that are in the map. First posted on arXiv 22 Feb 2023.

Builds on

39 of 50 references.

Closest ancestors · 19

Same direction, newest first

  1. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothing6 Jun
  2. 2022DARA: Dynamics-Aware Reward Augmentation in Offline Reinforcement Learning13 Mar
  3. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning23 Feb
  4. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learning5 Feb
  5. 2021RvS: What is Essential for Offline RL via Supervised Learning?20 Dec
  6. 2021Towards Hyperparameter-free Policy Selection for Offline Reinforcement Learning26 Oct
  7. 2021Offline Reinforcement Learning with Soft Behavior Regularization14 Oct
  8. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  9. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  10. 2021Offline RL Without Off-Policy Evaluation16 Jun
  11. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  12. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
7 more
  1. 2020Hyperparameter Selection for Offline Reinforcement Learning17 Jul
  2. 2020Critic Regularized Regression26 Jun
  3. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  5. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  6. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb
  7. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 20

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2015Trust Region Policy Optimizationcited by 2,503
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  5. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  6. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  7. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  8. 2017Constrained Policy Optimizationcited by 928
  9. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  10. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  11. 2019Reinforcement Learning in Healthcare: A Surveycited by 308
  12. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
8 more
  1. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
  2. 2018Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guaranteescited by 138
  3. 2018Learning to Navigate in Cities Without a Mapcited by 103
  4. 2019Wasserstein Adversarial Imitation Learningcited by 59
  5. 2021Generalized Proximal Policy Optimization with Sample Reusecited by 39
  6. 2021Learn Goal-Conditioned Policy with Intrinsic Motivation for Deep Reinforcement Learningcited by 15
  7. 2021Unsupervised Domain Adaptation with Dynamics-Aware Rewards in Reinforcement Learningcited by 15
  8. 2021Off-Dynamics Inverse Reinforcement Learning from Hetero-Domaincited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 12; 2024: 11; 2025: 9; 2026: 11.

43 papers.

Sort:

By area:

  1. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learningcited by 121
  2. 2025The Best Instruction-Tuning Data are Those That Fitcited by 48
  3. 2025RL-100: Performant Robotic Manipulation with Real-World Reinforcement Learningcited by 47
  4. 2023ChiPFormer: Transferable Chip Placement via Offline Decision Transformercited by 42
  5. 2023Beyond Reward: Offline Preference-guided Policy Optimizationcited by 35
  6. 2024Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learningcited by 33
  7. 2023PROTO: Iterative Policy Regularized Offline-to-Online Reinforcement Learningcited by 28
  8. 2024Wavelet Diffusion Neural Operatorcited by 28
  9. 2023Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learningcited by 27
  10. 2024Reinformer: Max-Return Sequence Modeling for Offline RLcited by 25
  11. 2023Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimizationcited by 23
  12. 2024Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Surveycited by 23
31 more
  1. 2024DiffPhyCon: A Generative Approach to Control Complex Physical Systemscited by 17
  2. 2023CEIL: Generalized Contextual Imitation Learningcited by 12
  3. 2023CLUE: Calibrated Latent Guidance for Offline Reinforcement Learningcited by 11
  4. 2024A2PO: Towards Effective Offline Reinforcement Learning from an Advantage-aware Perspectivecited by 10
  5. 2024Scalable and Effective Arithmetic Tree Generation for Adder and Multiplier Designscited by 9
  6. 2023A Simple Unified Uncertainty-Guided Framework for Offline-to-Online Reinforcement Learningcited by 8
  7. 2023Design from Policies: Conservative Test-Time Adaptation for Offline Policy Optimizationcited by 8
  8. 2024Exclusively Penalized Q-learning for Offline Reinforcement Learningcited by 8
  9. 2024CL-DiffPhyCon: Closed-loop Diffusion Control of Complex Physical Systemscited by 8
  10. 2023Making Offline RL Online: Collaborative World Models for Offline Visual Reinforcement Learningcited by 7
  11. 2025TDMPBC: Self-Imitative Reinforcement Learning for Humanoid Robot Controlcited by 6
  12. 2026Failure-Aware RL: Reliable Offline-to-Online Reinforcement Learning with Self-Recovery for Real-World Manipulationcited by 6
  13. 2024DIDI: Diffusion-Guided Diversity for Offline Behavioral Generationcited by 4
  14. 2023OER: Offline Experience Replay for Continual Offline Reinforcement Learningcited by 3
  15. 2024SelfBC: Self Behavior Cloning for Offline Reinforcement Learningcited by 3
  16. 2025VARD: Efficient and Dense Fine-Tuning for Diffusion Models with Value-based RLcited by 3
  17. 2026Self-evolving LLM agents with in-distribution Optimizationcited by 3
  18. 2025Boundary-to-Region Supervision for Offline Safe Reinforcement Learningcited by 1
  19. 2026Accelerated Sequential Flow Matching: A Bayesian Filtering Perspectivecited by 1
  20. 2026Noisy-Space Policy Gradient for Diffusion Policies in Offline Reinforcement Learningcited by 1
  21. 2025Representation Convergence: Mutual Distillation is Secretly a Form of Regularizationcited by 0
  22. 2025Sample-Efficient Diffusion-based Control of Complex Physics Systemscited by 0
  23. 2025CAWR: Corruption-Averse Advantage-Weighted Regression for Robust Policy Optimizationcited by 0
  24. 2025Symmetric Behavior Regularized Policy Optimizationcited by 0
  25. 2026ExO-PPO: an Extended Off-policy Proximal Policy Optimization Algorithmcited by 0
  26. 2026Mitigating Data Scarcity in Spaceflight Applications for Offline Reinforcement Learning Using Physics-Informed Deep Generative Modelscited by 0
  27. 2026Peng's Q(λ) for Conservative Value Estimation in Offline Reinforcement Learningcited by 0
  28. 2026COOPO: Cyclic Offline-Online Policy Optimization Algorithmcited by 0
  29. 2026Visuomotor Robotic Pruning in Planar Orchards Using Hybrid Reinforcement Learningcited by 0
  30. 2026From Weak Data to Strong Policy: Q-Targets Enable Provable In-Context Reinforcement Learningcited by 0
  31. 2026RL-PaO: Prediction as Action in Decision Making under Uncertaintycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2020MOPO: Model-based Offline Policy Optimization
  2. 2021You Only Evaluate Once: a Simple Baseline Algorithm for Offline RL
  3. 2023Iteratively Refined Behavior Regularization for Offline Reinforcement Learning
  4. 2022Behavior Prior Representation learning for Offline Reinforcement Learning
  5. 2022State-Aware Proximal Pessimistic Algorithms for Offline Reinforcement Learning
  6. 2022Offline Reinforcement Learning with Closed-Form Policy Improvement Operators
  7. 2025Behavior Preference Regression for Offline Reinforcement Learning
  8. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimation