mlsanityData to 14 Oct 2026

Constrained Policy Optimization via Sampling-Based Weight-Space Projection

arXiv 2512.13788 ↗First posted 15 Dec 2025cs.LG cs.ROcited by 0

On the map: Deep reinforcement learning methods → Safe and constrained RL

Abstract (arXiv)

Safety-critical learning requires policies that improve performance without leaving the safe operating regime. We study constrained policy learning where model parameters must satisfy rollout-based safety constraints that can be evaluated but not differentiated analytically. We propose SCPO, a sampling-based weight-space projection method that enforces safety directly in parameter space without requiring gradient access to the constraint functions. SCPO constructs a local safe region by combining rollout-based safety evaluations with smoothness bounds relating parameter perturbations to changes in safety metrics, and projects each gradient update via a convex QCQP. We establish a safe-by-induction guarantee: starting from any safe initialization, all intermediate policies remain safe given feasible projections. In constrained control settings with a stabilizing backup policy, SCPO further ensures closed-loop stability while enabling safe adaptation beyond the conservative backup. Experiments on constrained regression with harmful supervision and double-integrator imitation with a malicious expert show that SCPO rejects unsafe updates, maintains feasibility throughout training, and achieves meaningful objective improvement.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Safe and constrained RL, one of 15 directions in Deep reinforcement learning methods. Builds on 15 of its 24 references that are in the map. First posted on arXiv 15 Dec 2025.

Builds on

15 of 24 references.

References · 15

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2015Trust Region Policy Optimizationcited by 2,580
  4. 2017Constrained Policy Optimizationcited by 983
  5. 2018A Lyapunov-based Approach to Safe Reinforcement Learningcited by 297
  6. 2019Algorithms for Verifying Deep Neural Networkscited by 221
  7. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 181
  8. 2020CRPO: A New Approach for Safe Reinforcement Learning with Convergence Guaranteecited by 134
  9. 2024Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Modelscited by 125
  10. 2021Safe Reinforcement Learning Using Advantage-Based Interventioncited by 47
  11. 2025SaLoRA: Safety-Alignment Preserved Low-Rank Adaptationcited by 42
  12. 2020Safe Reinforcement Learning via Projection on a Safe Set: How to Achieve Optimality?cited by 34
3 more
  1. 2024HardNet: Hard-Constrained Neural Networks with Universal Approximation Guaranteescited by 31
  2. 2022Safety Correction from Baseline: Towards the Risk-aware Policy in Robotics via Dual-agent Reinforcement Learningcited by 3
  3. 2025A Simple Approach to Constraint-Aware Imitation Learning with Application to Autonomous Racingcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement Learning
  2. 2021Safe Continuous Control with Constrained Model-Based Policy Optimization
  3. 2024Safe Reinforcement Learning using Finite-Horizon Gradient-based Estimation
  4. 2024Absolute State-wise Constrained Policy Optimization: High-Probability State-wise Constraints Satisfaction
  5. 2025A Physics-Informed Machine Learning Framework for Safe and Optimal Control of Autonomous Systems
  6. 2025KFCPO: Kronecker-Factored Approximated Constrained Policy Optimization
  7. 2025SB-TRPO: Towards Safe Reinforcement Learning with Hard Constraints
  8. 2024Sampling-based Safe Reinforcement Learning for Nonlinear Dynamical Systems