mlsanityData to 14 Oct 2026

COP-Q: Safety-First Reinforcement Learning for Robot Control via Cholesky-Ordered Projection

arXiv 2606.04749 ↗First posted 3 Jun 2026cs.RO cs.LGcited by 0

On the map: Deep reinforcement learning methods → Safe and constrained RL

Abstract (arXiv)

Safe robot control requires maximizing return while satisfying safety constraints. In off-policy safe reinforcement learning, reward and safety Q-values are commonly learned by separate critic ensembles, with uncertainty handled independently for each objective. This objective-wise treatment neglects inter-objective correlation and can lead to overly conservative value estimates, thereby reducing sample efficiency. To address this issue, we propose Cholesky-Ordered Projection Q-learning (COP-Q), a safety-first method that incorporates inter-objective covariance into vector-valued Q-value estimation. COP-Q constructs a generalized confidence bound in the joint Q-value space and uses Cholesky factorization to encode objective priority in a sequential form. This preserves conservatism on safety while adaptively reducing excessive conservatism on the reward objective. The resulting estimate is used in both temporal-difference target computation and actor optimization. COP-Q incurs minimal computational overhead and is readily compatible with most existing deep Q-learning frameworks. Experiments on robot locomotion in Brax and safe navigation in Safety-Gymnasium, covering both hard- and soft-safety settings, demonstrate that COP-Q achieves strong safety performance together with competitive or improved sample efficiency relative to representative baselines.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Safe and constrained RL, one of 15 directions in Deep reinforcement learning methods. Builds on 23 of its 31 references that are in the map. First posted on arXiv 3 Jun 2026.

Builds on

23 of 31 references.

Closest ancestors · 2

Same direction, newest first

  1. 2026Off-Policy Safe Reinforcement Learning with Constrained Optimistic Exploration25 Mar
  2. 2024Off-Policy Primal-Dual Safe Reinforcement Learning26 Jan

Other references · 21

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  3. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  4. 2017Constrained Policy Optimizationcited by 983
  5. 2018Reward Constrained Policy Optimizationcited by 417
  6. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 407
  7. 2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 371
  8. 2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 316
  9. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
  10. 2021A Practical Guide to Multi-Objective Reinforcement Learning and Planningcited by 256
  11. 2023Safety-Gymnasium: A Unified Safe Reinforcement Learning Benchmarkcited by 143
  12. 2020Safe Reinforcement Learning in Constrained Markov Decision Processescited by 115
9 more
  1. 2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learningcited by 91
  2. 2023OmniSafe: An Infrastructure for Accelerating Safe Reinforcement Learning Researchcited by 80
  3. 2022Constrained Variational Policy Optimization for Safe Reinforcement Learningcited by 71
  4. 2022Saute RL: Almost Surely Safe Reinforcement Learning Using State Augmentationcited by 69
  5. 2022Constrained Update Projection Approach to Safe Policy Optimizationcited by 58
  6. 2023SafeDreamer: Safe Reinforcement Learning with World Modelscited by 54
  7. 2022A Review of Uncertainty for Deep Reinforcement Learningcited by 40
  8. 2022Safe Reinforcement Learning for Legged Locomotioncited by 35
  9. 2025Optimistic Exploration for Risk-Averse Constrained Reinforcement Learningcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Safe-Support Q-Learning: Learning without Unsafe Exploration
  2. 2026Learning to maintain safety through expert demonstrations in settings with unknown constraints: A Q-learning perspective
  3. 2024Q-learning-based Model-free Safety Filter
  4. 2025Safety-Aware Reinforcement Learning for Control via Risk-Sensitive Action-Value Iteration and Quantile Regression
  5. 2026Safe Execution of RL Policies Via Acceleration-Based CBF-QP Constraint Enforcement for Real-World Robotic Deployments
  6. 2026Sampling-Based Safe Reinforcement Learning
  7. 2026PPO-EAL: Exact Augmented Lagrangian Proximal Policy Optimization for Safe Robotic Control
  8. 2026Safe Flow Q-Learning: Offline Safe Reinforcement Learning with Reachability-Based Flow Policies