mlsanityData to 7 Oct 2026

Solving Multi-Agent Safe Optimal Control with Distributed Epigraph Form MARL

arXiv 2504.15425 ↗First posted 21 Apr 2025cs.RO cs.AI cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Tasks for multi-robot systems often require the robots to collaborate and complete a team goal while maintaining safety. This problem is usually formalized as a constrained Markov decision process (CMDP), which targets minimizing a global cost and bringing the mean of constraint violation below a user-defined threshold. Inspired by real-world robotic applications, we define safety as zero constraint violation. While many safe multi-agent reinforcement learning (MARL) algorithms have been proposed to solve CMDPs, these algorithms suffer from unstable training in this setting. To tackle this, we use the epigraph form for constrained optimization to improve training stability and prove that the centralized epigraph form problem can be solved in a distributed fashion by each agent. This results in a novel centralized training distributed execution MARL algorithm named Def-MARL. Simulation experiments on 8 different tasks across 2 different simulators show that Def-MARL achieves the best overall performance, satisfies safety constraints, and maintains stable training. Real-world hardware experiments on Crazyflie quadcopters demonstrate the ability of Def-MARL to safely coordinate agents to complete complex collaborative tasks compared to other methods.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 53 of its 91 references that are in the map. First posted on arXiv 21 Apr 2025.

Builds on

53 of 91 references.

References · 53

  1. 2015Trust Region Policy Optimizationcited by 2,503
  2. 2021Convex optimizationcited by 2,479
  3. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  4. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
  5. 2017Constrained Policy Optimizationcited by 928
  6. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 903
  7. 2017Counterfactual Multi-Agent Policy Gradientscited by 733
  8. 2019Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithmscited by 555
  9. 2020Masked Label Prediction: Unified Message Passing Model for Semi-Supervised Classificationcited by 500
  10. 2017Value-Decomposition Networks For Cooperative Multi-Agent Learningcited by 490
  11. 2020Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 475
  12. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 457
38 more
  1. 2018Reward Constrained Policy Optimizationcited by 397
  2. 2020QPLEX: Duplex Dueling Multi-Agent Q-Learningcited by 267
  3. 2020Weighted QMIX: Expanding Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 197
  4. 2018Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agentscited by 189
  5. 2017Hamilton-Jacobi Reachability: A Brief Overview and Recent Advancescited by 186
  6. 2020FACMAC: Factored Multi-Agent Centralised Policy Gradientscited by 178
  7. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 169
  8. 2017Socially Aware Motion Planning with Deep Reinforcement Learningcited by 128
  9. 2020CRPO: A New Approach for Safe Reinforcement Learning with Convergence Guaranteecited by 128
  10. 2016Decentralized Non-communicating Multiagent Collision Avoidance with Deep Reinforcement Learningcited by 104
  11. 2018Motion Planning Among Dynamic, Decision-Making Agents with Deep Reinforcement Learningcited by 101
  12. 2017Towards Optimally Decentralized Multi-Robot Collision Avoidance via Deep Reinforcement Learningcited by 81
  13. 2019Safe Reinforcement Learning Using Robust MPCcited by 72
  14. 2021Learning Safe Multi-Agent Control with Decentralized Neural Barrier Certificatescited by 71
  15. 2017Lifelong Multi-Agent Path Finding for Online Pickup and Delivery Taskscited by 62
  16. 2021Safe Multi-Agent Reinforcement Learning via Shieldingcited by 60
  17. 2022Reachability Constrained Reinforcement Learningcited by 52
  18. 2023SafeDreamer: Safe Reinforcement Learning with World Modelscited by 50
  19. 2019Learning Transferable Cooperative Behavior in Multi-Agent Teamscited by 40
  20. 2022Scalable Multi-Agent Reinforcement Learning through Intelligent Information Aggregationcited by 34
  21. 2020Constrained Markov Decision Processes via Backward Value Functionscited by 33
  22. 2023Iterative Reachability Estimation for Safe Reinforcement Learningcited by 33
  23. 2021Multi-Agent Constrained Policy Optimisationcited by 32
  24. 2023Solving Stabilize-Avoid Optimal Control via Epigraph Form and Deep Reinforcement Learningcited by 24
  25. 2020Multi-agent Motion Planning for Dense and Dynamic Environments via Deep Reinforcement Learningcited by 21
  26. 2024GCBF+: A Neural Graph Control Barrier Function Framework for Distributed Safe Multi-Agent Controlcited by 20
  27. 2019MAMPS: Safe Multi-Agent Reinforcement Learning via Model Predictive Shieldingcited by 17
  28. 2023AutoCost: Evolving Intrinsic Cost for Zero-violation Reinforcement Learningcited by 16
  29. 2024Solving Minimum-Cost Reach Avoid using Reinforcement Learningcited by 14
  30. 2019Online Trajectory Generation with Distributed Model Predictive Control for Multi-Robot Motion Planningcited by 13
  31. 2023Provably Efficient Generalized Lagrangian Policy Optimization for Safe Multi-Agent Reinforcement Learningcited by 13
  32. 2023Learning Safe Control for Multi-Robot Systems: Methods, Verification, and Open Challengescited by 12
  33. 2023Model-based Dynamic Shielding for Safe and Efficient Multi-Agent Reinforcement Learningcited by 11
  34. 2021Safe Multi-Agent Reinforcement Learning through Decentralized Multiple Control Barrier Functionscited by 9
  35. 2023Neural Graph Control Barrier Functions Guided Distributed Collision-avoidance Multi-agent Controlcited by 8
  36. 2021Safe Value Functionscited by 7
  37. 2025Discrete GCBF Proximal Policy Optimization for Multi-agent Safe Optimal Controlcited by 7
  38. 2018Trajectory Generation for Multiagent Point-To-Point Transitions via Distributed Model Predictive Controlcited by 4

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025Hierarchical Multi-Agent Reinforcement Learning with Control Barrier Functions for Safety-Critical Autonomous Systemscited by 4
  2. 2026How Does the Lagrangian Guide Safe Reinforcement Learning through Diffusion Models?cited by 4
  3. 2025MAD-PINN: A Decentralized Physics-Informed Machine Learning Framework for Safe and Optimal Multi-Agent Controlcited by 0
  4. 2026Solving Parameter-Robust Avoid Problems with Unknown Feasibility using Reinforcement Learningcited by 0