mlsanityData to 7 Oct 2026

Constrained Policy Optimization with Explicit Behavior Density for Offline Reinforcement Learning

arXiv 2301.12130 ↗First posted 28 Jan 2023cs.LGcited by 16

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Due to the inability to interact with the environment, offline reinforcement learning (RL) methods face the challenge of estimating the Out-of-Distribution (OOD) points. Existing methods for addressing this issue either control policy to exclude the OOD action or make the Q function pessimistic. However, these methods can be overly conservative or fail to identify OOD areas accurately. To overcome this problem, we propose a Constrained Policy optimization with Explicit Behavior density (CPED) method that utilizes a flow-GAN model to explicitly estimate the density of behavior policy. By estimating the explicit density, CPED can accurately identify the safe region and enable optimization within the region, resulting in less conservative learning policies. We further provide theoretical results for both the flow-GAN estimator and performance guarantee for CPED by showing that CPED can find the optimal Q-function value. Empirically, CPED outperforms existing alternatives on various standard offline reinforcement learning tasks, yielding higher expected returns.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 53 of its 85 references that are in the map. First posted on arXiv 28 Jan 2023.

Builds on

53 of 85 references.

Closest ancestors · 18

Same direction, newest first

  1. 2023Extreme Q-Learning: MaxEnt RL without Entropy5 Jan
  2. 2022Regularizing a Model-based Policy Stationary Distribution to Stabilize Offline Reinforcement Learning14 Jun
  3. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning9 Jun
  4. 2022Latent-Variable Advantage-Weighted Policy Optimization for Offline RL16 Mar
  5. 2022A Behavior Regularized Implicit Policy for Offline Reinforcement Learning19 Feb
  6. 2022Supported Policy Optimization for Offline Reinforcement Learning13 Feb
  7. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  8. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  9. 2021Offline RL Without Off-Policy Evaluation16 Jun
  10. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  11. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning17 May
  12. 2021Risk-Averse Offline Reinforcement Learning10 Feb
6 more
  1. 2020PLAS: Latent Action Space for Offline Reinforcement Learning14 Nov
  2. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL21 Jul
  3. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  5. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  6. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 35

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2015Continuous control with deep reinforcement learningcited by 3,279
  5. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  7. 2017Improved Training of Wasserstein GANscited by 2,223
  8. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,048
  9. 2016Density estimation using Real NVPcited by 2,029
  10. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  11. 2016Generative Adversarial Imitation Learningcited by 1,576
  12. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
23 more
  1. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  2. 2014NICE: Non-linear Independent Components Estimationcited by 1,052
  3. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  4. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  5. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  6. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  7. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  8. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  9. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
  10. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
  11. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  12. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 268
  13. 2017MMD GAN: Towards Deeper Understanding of Moment Matching Networkcited by 251
  14. 2021Reinforcement learning based recommender systems: A surveycited by 187
  15. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 177
  16. 2016A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Modelscited by 167
  17. 2020Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Criticscited by 154
  18. 2020Parrot: Data-Driven Behavioral Priors for Reinforcement Learningcited by 136
  19. 2018How Well Generative Adversarial Networks Learn Distributionscited by 78
  20. 2016Model-Free Imitation Learning with Policy Optimizationcited by 52
  21. 2017Approximation and Convergence Properties of Generative Adversarial Learningcited by 28
  22. 2018GAN Q-learningcited by 7
  23. 2020High-Dimensional Non-Parametric Density Estimation in Mixed Smooth Sobolev Spacescited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 5; 2025: 6; 2026: 4.

16 papers.

Sort:

By area:

  1. 2024Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directionscited by 20
  2. 2024FlowPG: Action-constrained Policy Gradient with Normalizing Flowscited by 16
  3. 2025GenPO: Generative Diffusion Models Meet On-Policy Reinforcement Learningcited by 16
  4. 2025A Survey of Safe Reinforcement Learning and Constrained MDPs: A Technical Survey on Single-Agent and Multi-Agent Safetycited by 12
  5. 2023Iteratively Refined Behavior Regularization for Offline Reinforcement Learningcited by 10
  6. 2024Q-Distribution guided Q-learning for offline reinforcement learning: Uncertainty penalized Q-value via consistency modelcited by 4
  7. 2026ReFORM: Reflected Flows for On-support Offline RL via Noise Manipulationcited by 4
  8. 2024Augmenting Offline Reinforcement Learning with State-only Interactionscited by 2
  9. 2025Adaptive Neighborhood-Constrained Q Learning for Offline Reinforcement Learningcited by 2
  10. 2025Guardian-regularized Safe Offline Reinforcement Learning for Smart Weaning of Mechanical Circulatory Devicescited by 2
  11. 2024DiffPoGAN: Diffusion Policies with Generative Adversarial Networks for Offline Reinforcement Learningcited by 0
  12. 2025Policy Constraint by Only Support Constraint for Offline Reinforcement Learningcited by 0
4 more
  1. 2025Quantile Q-Learning: Revisiting Offline Extreme Q-Learning with Quantile Regressioncited by 0
  2. 2026LexiSafe: Offline Safe Reinforcement Learning with Lexicographic Safety-Reward Hierarchycited by 0
  3. 2026Generative OOD-regularized Model-based Policy Optimizationcited by 0
  4. 2026Improving Offline Goal-Conditioned Reinforcement Learning via Selective Reward Stimulationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Preferred-Action-Optimized Diffusion Policies for Offline Reinforcement Learning
  2. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimation
  3. 2024Offline Behavior Distillation
  4. 2024Constrained Latent Action Policies for Model-Based Offline Reinforcement Learning
  5. 2022Offline Reinforcement Learning with Closed-Form Policy Improvement Operators
  6. 2026Diffusion Policy with Behavioral Advantage Correction for Offline Reinforcement Learning
  7. 2024Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning
  8. 2020MOPO: Model-based Offline Policy Optimization