mlsanityData to 7 Oct 2026

Constrained Variational Policy Optimization for Safe Reinforcement Learning

arXiv 2201.11927 ↗First posted 28 Jan 2022cs.LG cs.AI cs.ROcited by 69

Not on the map. Why?

Abstract (arXiv)

Safe reinforcement learning (RL) aims to learn policies that satisfy certain constraints before deploying them to safety-critical applications. Previous primal-dual style approaches suffer from instability issues and lack optimality guarantees. This paper overcomes the issues from the perspective of probabilistic inference. We introduce a novel Expectation-Maximization approach to naturally incorporate constraints during the policy learning: 1) a provable optimal non-parametric variational distribution could be computed in closed form after a convex optimization (E-step); 2) the policy parameter is improved within the trust region based on the optimal variational distribution (M-step). The proposed algorithm decomposes the safe RL problem into a convex optimization phase and a supervised learning phase, which yields a more stable training performance. A wide range of experiments on continuous robotic tasks shows that the proposed method achieves significantly better constraint satisfaction performance and better sample efficiency than baselines. The code is available at https://github.com/liuzuxin/cvpo-safe-rl.

Key facts. Cited by 69 papers in the map (data through 7 Oct 2026). Builds on 31 of its 46 references that are in the map. First posted on arXiv 28 Jan 2022.

Builds on

31 of 46 references.

References · 31

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  5. 2015Trust Region Policy Optimizationcited by 2,503
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  7. 2017Constrained Policy Optimizationcited by 928
  8. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  9. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  10. 2017Safe Reinforcement Learning via Shieldingcited by 427
  11. 2018Reward Constrained Policy Optimizationcited by 397
  12. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
19 more
  1. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 320
  2. 2018Maximum a Posteriori Policy Optimisationcited by 318
  3. 2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 302
  4. 2018A Lyapunov-based Approach to Safe Reinforcement Learningcited by 278
  5. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 260
  6. 2019End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Taskscited by 256
  7. 2018Safe Exploration in Continuous Action Spacescited by 237
  8. 2020Projection-Based Constrained Policy Optimizationcited by 195
  9. 2020Recovery RL: Safe Reinforcement Learning with Learned Recovery Zonescited by 156
  10. 2020CRPO: A New Approach for Safe Reinforcement Learning with Convergence Guaranteecited by 128
  11. 2017Trial without Error: Towards Safe Reinforcement Learning via Human Interventioncited by 102
  12. 2019V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Controlcited by 85
  13. 2018Accelerated Primal-Dual Policy Optimization for Safe Reinforcement Learningcited by 61
  14. 2022Constrained Policy Optimization via Bayesian World Modelscited by 55
  15. 2021Safe Reinforcement Learning Using Advantage-Based Interventioncited by 45
  16. 2019Value constrained model-free continuous controlcited by 34
  17. 2021Context-Aware Safe Reinforcement Learning for Non-Stationary Environmentscited by 33
  18. 2018Relative Entropy Regularized Policy Iterationcited by 32
  19. 2018VIREL: A Variational Inference Framework for Reinforcement Learningcited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 19; 2024: 16; 2025: 11; 2026: 18.

69 papers.

Sort:

By area:

  1. 2023Constrained Decision Transformer for Offline Safe Reinforcement Learningcited by 65
  2. 2023Datasets and Benchmarks for Offline Safe Reinforcement Learningcited by 58
  3. 2024A Survey of Constraint Formulations in Safe Reinforcement Learningcited by 42
  4. 2022On the Robustness of Safe Reinforcement Learning under Observational Perturbationscited by 38
  5. 2022Trustworthy Reinforcement Learning Against Intrinsic Vulnerabilities: Robustness, Safety, and Generalizabilitycited by 32
  6. 2023Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learningcited by 27
  7. 2023Safe Offline Reinforcement Learning with Real-Time Budget Constraintscited by 26
  8. 2024ActSafe: Active Exploration with Safety Constraints for Reinforcement Learningcited by 22
  9. 2023Trust Region-Based Safe Distributional Reinforcement Learning for Multiple Constraintscited by 20
  10. 2023Constraint-Conditioned Policy Optimization for Versatile Safe Reinforcement Learningcited by 17
  11. 2023Safety-aware Causal Representation for Trustworthy Offline Reinforcement Learning in Autonomous Drivingcited by 16
  12. 2023Learning Shared Safety Constraints from Multi-task Demonstrationscited by 15
38 more
  1. 2023Guided Online Distillation: Promoting Safe Reinforcement Learning by Offline Demonstrationcited by 14
  2. 2024Off-Policy Primal-Dual Safe Reinforcement Learningcited by 13
  3. 2023Risk-Averse Model Uncertainty for Distributionally Robust Safe Reinforcement Learningcited by 12
  4. 2023A Survey of Demonstration Learningcited by 12
  5. 2026BAGEN: Are LLM Agents Budget-Aware?cited by 12
  6. 2023Efficient Off-Policy Safe Reinforcement Learning Using Trust Region Conditional Value at Riskcited by 11
  7. 2023Safe Reinforcement Learning with Dual Robustnesscited by 10
  8. 2024OASIS: Conditional Distribution Shaping for Offline Safe Reinforcement Learningcited by 10
  9. 2026Machine Unlearning under Retain-Forget Entanglementcited by 10
  10. 2023Optimal Transport Perturbations for Safe Reinforcement Learning with Robustness Guaranteescited by 9
  11. 2024Variational Delayed Policy Optimizationcited by 9
  12. 2022The Real Deal: A Review of Challenges and Opportunities in Moving Reinforcement Learning-Based Traffic Signal Control Systems Towards Realitycited by 8
  13. 2023Gradient Shaping for Multi-Constraint Safe Reinforcement Learningcited by 7
  14. 2024Latent Safety-Constrained Policy Approach for Safe Offline Reinforcement Learningcited by 7
  15. 2024Reinforcement Learning with Adaptive Regularization for Safe Control of Critical Systemscited by 6
  16. 2024Safe Reinforcement Learning in Black-Box Environments via Adaptive Shieldingcited by 6
  17. 2026Safe Exploration via Policy Priorscited by 6
  18. 2022Effects of Safety State Augmentation on Safe Explorationcited by 5
  19. 2023Reduced Policy Optimization for Continuous Control with Hard Constraintscited by 5
  20. 2024Knowledge Transfer from Simple to Complex: A Safe and Efficient Reinforcement Learning Framework for Autonomous Driving Decision-Makingcited by 5
  21. 2024SPO: Sequential Monte Carlo Policy Optimisationcited by 4
  22. 2025Advancing Autonomous VLM Agents via Variational Subgoal-Conditioned Reinforcement Learningcited by 4
  23. 2025Safety Representations for Safer Policy Learningcited by 4
  24. 2026How Does the Lagrangian Guide Safe Reinforcement Learning through Diffusion Models?cited by 4
  25. 2024Feasibility Consistent Representation Learning for Safe Reinforcement Learningcited by 3
  26. 2025Optimistic Exploration for Risk-Averse Constrained Reinforcement Learningcited by 3
  27. 2024Federated Reinforcement Learning with Constraint Heterogeneitycited by 2
  28. 2024Safe Reinforcement Learning using Finite-Horizon Gradient-based Estimationcited by 2
  29. 2025PNAct: Crafting Backdoor Attacks in Safe Reinforcement Learningcited by 2
  30. 2026Off-Policy Safe Reinforcement Learning with Constrained Optimistic Explorationcited by 2
  31. 2026SHAPO: Sharpness-Aware Policy Optimization for Safe Explorationcited by 2
  32. 2022Finding Safe Zones of policies Markov Decision Processescited by 1
  33. 2023Constrained Proximal Policy Optimizationcited by 1
  34. 2023Probabilistic Constrained Reinforcement Learning with Formal Interpretabilitycited by 1
  35. 2024Constrained Optimal Fuel Consumption of HEV: A Constrained Reinforcement Learning Approachcited by 1
  36. 2024Safe PDE Boundary Control with Neural Operatorscited by 1
  37. 2025QuietPaw: Learning Quadrupedal Locomotion with Versatile Noise Preference Alignmentcited by 1
  38. 2025Bidirectional Soft Actor-Critic: Leveraging Forward and Reverse KL Divergence for Efficient Reinforcement Learningcited by 1

50 most cited of 69 All 69 (JSON)