mlsanityData to 7 Oct 2026

Trust Region-Based Safe Distributional Reinforcement Learning for Multiple Constraints

arXiv 2301.10923 ↗First posted 26 Jan 2023cs.LG cs.AIcited by 20

Not on the map. Why?

Abstract (arXiv)

In safety-critical robotic tasks, potential failures must be reduced, and multiple constraints must be met, such as avoiding collisions, limiting energy consumption, and maintaining balance. Thus, applying safe reinforcement learning (RL) in such robotic tasks requires to handle multiple constraints and use risk-averse constraints rather than risk-neutral constraints. To this end, we propose a trust region-based safe RL algorithm for multiple constraints called a safe distributional actor-critic (SDAC). Our main contributions are as follows: 1) introducing a gradient integration method to manage infeasibility issues in multi-constrained problems, ensuring theoretical convergence, and 2) developing a TD(λ) target distribution to estimate risk-averse constraints with low biases. We evaluate SDAC through extensive experiments involving multi- and single-constrained robotic tasks. While maintaining high scores, SDAC shows 1.93 times fewer steps to satisfy all constraints in multi-constrained tasks and 1.78 times fewer constraint violations in single-constrained tasks compared to safe RL baselines. Code is available at: https://github.com/rllab-snu/Safe-Distributional-Actor-Critic.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 31 of its 49 references that are in the map. First posted on arXiv 26 Jan 2023.

Builds on

31 of 49 references.

References · 31

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2015Trust Region Policy Optimizationcited by 2,503
  3. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
  4. 2017Constrained Policy Optimizationcited by 928
  5. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  6. 2020Learning Quadrupedal Locomotion over Challenging Terraincited by 494
  7. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  8. 2021Conflict-Averse Gradient Descent for Multi-task Learningcited by 441
  9. 2021Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learningcited by 395
  10. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
  11. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 320
  12. 2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 302
19 more
  1. 2022Learning robust perceptive locomotion for quadrupedal robots in the wildcited by 299
  2. 2022Multi-Task Learning as a Bargaining Gamecited by 209
  3. 2020Projection-Based Constrained Policy Optimizationcited by 195
  4. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 169
  5. 2020Recovery RL: Safe Reinforcement Learning with Learned Recovery Zonescited by 156
  6. 2020Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Criticscited by 154
  7. 2020CRPO: A New Approach for Safe Reinforcement Learning with Convergence Guaranteecited by 128
  8. 2019IPO: Interior-point Policy Optimization under Constraintscited by 119
  9. 2020Conservative Safety Critics for Explorationcited by 107
  10. 2022Constrained Variational Policy Optimization for Safe Reinforcement Learningcited by 69
  11. 2022COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimationcited by 69
  12. 2018An Analysis of Categorical Distributional Reinforcement Learningcited by 68
  13. 2021Achieving Zero Constraint Violation for Constrained Reinforcement Learning via Primal-Dual Approachcited by 46
  14. 2018Feedback Control For Cassie With Deep Reinforcement Learningcited by 36
  15. 2021Density Constrained Reinforcement Learningcited by 25
  16. 2021GMAC: A Distributional Perspective on Actor-Critic Frameworkcited by 15
  17. 2021Safe Distributional Reinforcement Learningcited by 10
  18. 2022The Nature of Temporal Difference Errors in Multi-step Distributional Reinforcement Learningcited by 9
  19. 2022Conservative Distributional Reinforcement Learning with Safety Constraintscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 10; 2025: 5; 2026: 4.

20 papers.

Sort:

By area:

  1. 2024Distributionally Robust Constrained Reinforcement Learning under Strong Dualitycited by 15
  2. 2024OASIS: Conditional Distribution Shaping for Offline Safe Reinforcement Learningcited by 10
  3. 2024Safety Filters for Black-Box Dynamical Systems by Learning Discriminating Hyperplanescited by 8
  4. 2024Enhancing Efficiency of Safe Reinforcement Learning via Sample Manipulationcited by 8
  5. 2025One Subgoal at a Time: Zero-Shot Generalization to Arbitrary Linear Temporal Logic Requirements in Multi-Task Reinforcement Learningcited by 8
  6. 2025SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfercited by 8
  7. 2023Gradient Shaping for Multi-Constraint Safe Reinforcement Learningcited by 7
  8. 2024Constrained Reinforcement Learning Under Model Mismatchcited by 6
  9. 2024Spectral-Risk Safe Reinforcement Learning with Convergence Guaranteescited by 5
  10. 2024An Offline Adaptation Framework for Constrained Multi-Objective Reinforcement Learningcited by 5
  11. 2024Conflict-Averse Gradient Aggregation for Constrained Multi-Objective Reinforcement Learningcited by 4
  12. 2025Rectified Robust Policy Optimization for Model-Uncertain Constrained Reinforcement Learning without Strong Dualitycited by 4
8 more
  1. 2025Optimistic Exploration for Risk-Averse Constrained Reinforcement Learningcited by 3
  2. 2024Safe CoR: A Dual-Expert Approach to Integrating Imitation Learning and Safe Reinforcement Learning Using Constraint Rewardscited by 1
  3. 2026Federated Distributional Reinforcement Learning with Distributional Critic Regularizationcited by 1
  4. 2024Bellman Unbiasedness: Toward Provably Efficient Distributional Reinforcement Learning with General Value Function Approximationcited by 0
  5. 2025Don't Trade Off Safety: Diffusion Regularization for Constrained Offline RLcited by 0
  6. 2026Safe-Support Q-Learning: Learning without Unsafe Explorationcited by 0
  7. 2026Lyapunov-Guided Self-Alignment: Test-Time Adaptation for Offline Safe Reinforcement Learningcited by 0
  8. 2026Towards Efficient and Expressive Offline RL via Flow-Anchored Noise-conditioned Q-Learningcited by 0