mlsanityData to 7 Oct 2026

Utility-Constrained Policy Optimization

arXiv 2606.14029 ↗First posted 12 Jun 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Constrained MDPs (CMDPs) are a widely adopted framework for incorporating safety into RL agents; however, the framework does not support risk-sensitive constraints. This can be problematic: For example, CMDPs allow for optimal solutions that, in order to satisfy the risk-neutral constraints, mix infrequent catastrophic behaviors and frequent, overly conservative ones. Moreover, prior empirical results suggest that enforcing stricter, risk-sensitive constraints can improve performance even under risk-neutral evaluation. The natural framework to incorporate risk-sensitive constraints is utility-constrained MDPs (UCMDPs), but no practical solutions for this problem existed. In this work, we introduce a simple yet powerful methodology for UCMDPs and constrained RL. Besides allowing for risk-sensitive constraints, our framework does not require us to fix constraint limits in advance of training the agent, provided that a sensible range is known. This increases policy flexibility and, in practice, allows for adjustments to these limits at no extra training cost. Besides benefiting from the generality of the framework, our agent shows strong performance in practice, consistently matching or outperforming existing baselines in several Safety Gymnasium benchmark tasks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 37 of its 50 references that are in the map. First posted on arXiv 12 Jun 2026.

Builds on

37 of 50 references.

References · 37

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2021Convex optimizationcited by 2,479
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  5. 2017Constrained Policy Optimizationcited by 928
  6. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  7. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  8. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  9. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  10. 2018Reward Constrained Policy Optimizationcited by 397
  11. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 320
  12. 2018Maximum a Posteriori Policy Optimisationcited by 318
25 more
  1. 2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 302
  2. 2018A Lyapunov-based Approach to Safe Reinforcement Learningcited by 278
  3. 2020Projection-Based Constrained Policy Optimizationcited by 195
  4. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 145
  5. 2023Safety-Gymnasium: A Unified Safe Reinforcement Learning Benchmarkcited by 135
  6. 2020CRPO: A New Approach for Safe Reinforcement Learning with Convergence Guaranteecited by 128
  7. 2023OmniSafe: An Infrastructure for Accelerating Safe Reinforcement Learning Researchcited by 76
  8. 2022Constrained Variational Policy Optimization for Safe Reinforcement Learningcited by 69
  9. 2022Saute RL: Almost Surely Safe Reinforcement Learning Using State Augmentationcited by 63
  10. 2023SafeDreamer: Safe Reinforcement Learning with World Modelscited by 50
  11. 2020Learning Off-Policy with Online Planningcited by 46
  12. 2020Constrained Markov Decision Processes via Backward Value Functionscited by 33
  13. 2021Distributional Reinforcement Learning for Multi-Dimensional Reward Functionscited by 21
  14. 2018Distributional Multivariate Policy Evaluation and Exploration with the Bellman GANcited by 16
  15. 2024Foundations of Multivariate Distributional Reinforcement Learningcited by 16
  16. 2021Safe Exploration by Solving Early Terminated MDPcited by 13
  17. 2024Off-Policy Primal-Dual Safe Reinforcement Learningcited by 13
  18. 2023Beyond Average Return in Markov Decision Processescited by 11
  19. 2022Quantile Constrained Reinforcement Learning: A Reinforcement Learning Framework Constraining Outage Probabilitycited by 9
  20. 2025Optimizing Return Distributions with Distributional Dynamic Programmingcited by 9
  21. 2022Effects of Safety State Augmentation on Safe Explorationcited by 5
  22. 2024Spectral-Risk Safe Reinforcement Learning with Convergence Guaranteescited by 5
  23. 2025Optimistic Exploration for Risk-Averse Constrained Reinforcement Learningcited by 3
  24. 2026Extreme Value Policy Optimization for Safe Reinforcement Learningcited by 3
  25. 2026Off-Policy Safe Reinforcement Learning with Constrained Optimistic Explorationcited by 2

Built on by

Not cited yet.