mlsanityData to 14 Oct 2026

Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning

arXiv 2405.19909 ↗First posted 30 May 2024cs.LG cs.AI cs.ROcited by 13

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

In offline reinforcement learning, the challenge of out-of-distribution (OOD) is pronounced. To address this, existing methods often constrain the learned policy through policy regularization. However, these methods often suffer from the issue of unnecessary conservativeness, hampering policy improvement. This occurs due to the indiscriminate use of all actions from the behavior policy that generates the offline dataset as constraints. The problem becomes particularly noticeable when the quality of the dataset is suboptimal. Thus, we propose Adaptive Advantage-guided Policy Regularization (A2PR), obtaining high-advantage actions from an augmented behavior policy combined with VAE to guide the learned policy. A2PR can select high-advantage actions that differ from those present in the dataset, while still effectively maintaining conservatism from OOD actions. This is achieved by harnessing the VAE capacity to generate samples matching the distribution of the data points. We theoretically prove that the improvement of the behavior policy is guaranteed. Besides, it effectively mitigates value overestimation with a bounded performance gap. Empirically, we conduct a series of experiments on the D4RL benchmark, where A2PR demonstrates state-of-the-art performance. Furthermore, experimental results on additional suboptimal mixed datasets reveal that A2PR exhibits superior performance. Code is available at https://github.com/ltlhuuu/A2PR.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 35 of its 49 references that are in the map. First posted on arXiv 30 May 2024.

Builds on

35 of 49 references.

Closest ancestors · 16

Same direction, newest first

  1. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets6 Oct
  2. 2023Offline Reinforcement Learning with Imbalanced Datasets6 Jul
  3. 2023Harnessing Mixed Offline Reinforcement Learning Datasets via Trajectory Weighting22 Jun
  4. 2023Policy Regularization with Dataset Constraint for Offline Reinforcement Learning11 Jun
  5. 2023Boosting Offline Reinforcement Learning with Action Preference Query6 Jun
  6. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Library13 Oct
  7. 2022Latent-Variable Advantage-Weighted Policy Optimization for Offline RL16 Mar
  8. 2022Supported Policy Optimization for Offline Reinforcement Learning13 Feb
  9. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  10. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  11. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  12. 2020PLAS: Latent Action Space for Offline Reinforcement Learning14 Nov
4 more
  1. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  3. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  4. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 19

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015Trust Region Policy Optimizationcited by 2,580
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  5. 2016OpenAI Gymcited by 1,736
  6. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  7. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  8. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  9. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  10. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  11. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  12. 2018Learning Dexterous In-Hand Manipulationcited by 687
7 more
  1. 2019Learning agile and dynamic motor skills for legged robotscited by 504
  2. 2018Regularisation of Neural Networks by Enforcing Lipschitz Continuitycited by 314
  3. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 284
  4. 2022Mastering the Game of Stratego with Model-Free Multiagent Reinforcement Learningcited by 131
  5. 2022Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrationscited by 89
  6. 2022Versatile Offline Imitation from Observations and Examples via Regularized State-Occupancy Matchingcited by 43
  7. 2023Off-Policy Average Reward Actor-Critic with Deterministic Policy Searchcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 8; 2026: 3.

13 papers.

Sort:

By area:

  1. 2024Highly Efficient Self-Adaptive Reward Shaping for Reinforcement Learningcited by 22
  2. 2024Scaling Offline Model-Based RL via Jointly-Optimized World-Action Model Pretrainingcited by 14
  3. 2025Skill Expansion and Composition in Parameter Spacecited by 14
  4. 2025Dual-Robust Cross-Domain Offline Reinforcement Learning Against Dynamics Shiftscited by 6
  5. 2025Active Advantage-Aligned Online Reinforcement Learning with Offline Datacited by 5
  6. 2026ReFORM: Reflected Flows for On-support Offline RL via Noise Manipulationcited by 4
  7. 2025ADG: Ambient Diffusion-Guided Dataset Recovery for Corruption-Robust Offline Reinforcement Learningcited by 2
  8. 2025Efficient Cross-Domain Offline Reinforcement Learning with Dynamics- and Value-Aligned Data Filteringcited by 2
  9. 2025Adaptive Scaling of Policy Constraints for Offline Reinforcement Learningcited by 1
  10. 2025Diffusion Policies with Value-Conditional Optimization for Offline Reinforcement Learningcited by 1
  11. 2026Beyond Penalization: Diffusion-based Out-of-Distribution Detection and Selective Regularization in Offline Reinforcement Learningcited by 1
  12. 2025CAWR: Corruption-Averse Advantage-Weighted Regression for Robust Policy Optimizationcited by 0
1 more
  1. 2026Role-Adaptive Policy Optimization for Offline Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A2PO: Towards Effective Offline Reinforcement Learning from an Advantage-aware Perspective
  2. 2023Iteratively Refined Behavior Regularization for Offline Reinforcement Learning
  3. 2022Offline Reinforcement Learning with Adaptive Behavior Regularization
  4. 2022A Behavior Regularized Implicit Policy for Offline Reinforcement Learning
  5. 2023Offline Reinforcement Learning with On-Policy Q-Function Regularization
  6. 2025Taming OOD Actions for Offline Reinforcement Learning: An Advantage-Based Approach
  7. 2025Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RL
  8. 2025Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learning