mlsanityData to 14 Oct 2026

Safe and Balanced: A Framework for Constrained Multi-Objective Reinforcement Learning

arXiv 2405.16390 ↗First posted 26 May 2024cs.AI cs.LGcited by 9

On the map: Deep reinforcement learning methods → Safe and constrained RL

Abstract (arXiv)

In numerous reinforcement learning (RL) problems involving safety-critical systems, a key challenge lies in balancing multiple objectives while simultaneously meeting all stringent safety constraints. To tackle this issue, we propose a primal-based framework that orchestrates policy optimization between multi-objective learning and constraint adherence. Our method employs a novel natural policy gradient manipulation method to optimize multiple RL objectives and overcome conflicting gradients between different tasks, since the simple weighted average gradient direction may not be beneficial for specific tasks' performance due to misaligned gradients of different task objectives. When there is a violation of a hard constraint, our algorithm steps in to rectify the policy to minimize this violation. We establish theoretical convergence and constraint violation guarantees in a tabular setting. Empirically, our proposed method also outperforms prior state-of-the-art methods on challenging safe multi-objective reinforcement learning tasks.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in Safe and constrained RL, one of 15 directions in Deep reinforcement learning methods. Builds on 26 of its 57 references that are in the map. First posted on arXiv 26 May 2024.

Builds on

26 of 57 references.

References · 26

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2015Trust Region Policy Optimizationcited by 2,580
  3. 2020Gradient Surgery for Multi-Task Learningcited by 1,256
  4. 2017Constrained Policy Optimizationcited by 983
  5. 2018DeepMind Control Suitecited by 956
  6. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
  7. 2021Conflict-Averse Gradient Descent for Multi-task Learningcited by 459
  8. 2018Reward Constrained Policy Optimizationcited by 417
  9. 2018A Lyapunov-based Approach to Safe Reinforcement Learningcited by 297
  10. 2018A Finite Time Analysis of Temporal Difference Learning With Linear Function Approximationcited by 229
  11. 2022Multi-Task Learning as a Bargaining Gamecited by 215
  12. 2020Projection-Based Constrained Policy Optimizationcited by 203
14 more
  1. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 181
  2. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 164
  3. 2019Lyapunov-based Safe Policy Optimization for Continuous Controlcited by 145
  4. 2020CRPO: A New Approach for Safe Reinforcement Learning with Convergence Guaranteecited by 134
  5. 2019IPO: Interior-point Policy Optimization under Constraintscited by 132
  6. 2019Global Convergence of Policy Gradient Methods to (Almost) Locally Optimal Policiescited by 124
  7. 2018Learning-based Model Predictive Control for Safe Explorationcited by 120
  8. 2023OmniSafe: An Infrastructure for Accelerating Safe Reinforcement Learning Researchcited by 80
  9. 2016Safe Exploration in Finite Markov Decision Processes with Gaussian Processescited by 75
  10. 2022In Defense of the Unitary Scalarization for Deep Multi-Task Learningcited by 67
  11. 2020Reinforcement Learning in Economics and Financecited by 58
  12. 2020A Distributional View on Multi-Objective Policy Optimizationcited by 52
  13. 2019Temporal Logic Guided Safe Reinforcement Learning Using Control Barrier Functionscited by 21
  14. 2023A Human-Centered Safe Robot Reinforcement Learning Framework with Interactive Behaviorscited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 3.

9 papers.

  1. 2025Learning to Optimize Multi-Objective Alignment Through Dynamic Reward Weightingcited by 23
  2. 2025Robust Gymnasium: A Unified Modular Benchmark for Robust Reinforcement Learningcited by 13
  3. 2025A Survey of Safe Reinforcement Learning and Constrained MDPs: A Technical Survey on Single-Agent and Multi-Agent Safetycited by 13
  4. 2025Scalable Multi-Objective Robot Reinforcement Learning through Gradient Conflict Resolutioncited by 5
  5. 2025RLBenchNet: The Right Network for the Right Reinforcement Learning Taskcited by 2
  6. 2026Uncovering Cross-Objective Interference in Multi-Objective Alignmentcited by 2
  7. 2025LacaDM: A Latent Causal Diffusion Model for Multiobjective Reinforcement Learningcited by 1
  8. 2026Heterogeneous Knowledge Distillation via Geometry Decoupling and Momentum-Aware Gradient Regulationcited by 0
  9. 2026PROMO: Preference-conditioned Multi-Objective Reinforcement Learning for Quadrupedal Robotscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Conflict-Averse Gradient Aggregation for Constrained Multi-Objective Reinforcement Learning
  2. 2024A Survey of Constraint Formulations in Safe Reinforcement Learning
  3. 2024An Offline Adaptation Framework for Constrained Multi-Objective Reinforcement Learning
  4. 2024Safety Optimized Reinforcement Learning via Multi-Objective Policy Optimization
  5. 2024Balance Reward and Safety Optimization for Safe Reinforcement Learning: A Perspective of Gradient Manipulation
  6. 2026CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement Learning
  7. 2025SB-TRPO: Towards Safe Reinforcement Learning with Hard Constraints
  8. 2021On Multi-objective Policy Optimization as a Tool for Reinforcement Learning: Case Studies in Offline RL and Finetuning