# Safe and Generalizable Hierarchical Multi-Agent RL via Constraint Manifold Control

arXiv: https://arxiv.org/abs/2606.24010 · HTML: https://mlsanity.com/paper/2606.24010/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 38 of its 56 references that are in the map.
- First posted on arXiv 22 Jun 2026.

## Builds on (most cited first, 30 of 38)

- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017)
- [The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games](https://mlsanity.com/paper/2103.01955/) (2021)
- [Data-Efficient Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1805.08296/) (2018)
- [Reward Constrained Policy Optimization](https://mlsanity.com/paper/1805.11074/) (2018)
- [QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1905.05408/) (2019)
- [A Lyapunov-based Approach to Safe Reinforcement Learning](https://mlsanity.com/paper/1805.07708/) (2018)
- [End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks](https://mlsanity.com/paper/1903.08792/) (2019)
- [Accelerating Reinforcement Learning with Learned Skill Priors](https://mlsanity.com/paper/2010.11944/) (2020)
- [Lyapunov-based Safe Policy Optimization for Continuous Control](https://mlsanity.com/paper/1901.10031/) (2019)
- [CRPO: A New Approach for Safe Reinforcement Learning with Convergence Guarantee](https://mlsanity.com/paper/2011.05869/) (2020)
- [IPO: Interior-point Policy Optimization under Constraints](https://mlsanity.com/paper/1910.09615/) (2019)
- [A Deep Hierarchical Approach to Lifelong Learning in Minecraft](https://mlsanity.com/paper/1604.07255/) (2016)
- [The Safety Filter: A Unified View of Safety-Critical Control in Autonomous Systems](https://mlsanity.com/paper/2309.05837/) (2023)
- [Safe Reinforcement Learning Using Robust MPC](https://mlsanity.com/paper/1906.04005/) (2019)
- [Safe Reinforcement Learning Using Robust Control Barrier Functions](https://mlsanity.com/paper/2110.05415/) (2021)
- [SafeDreamer: Safe Reinforcement Learning with World Models](https://mlsanity.com/paper/2307.07176/) (2023)
- [Iterative Reachability Estimation for Safe Reinforcement Learning](https://mlsanity.com/paper/2309.13528/) (2023)
- [Scalable Multi-Agent Reinforcement Learning through Intelligent Information Aggregation](https://mlsanity.com/paper/2211.02127/) (2022)
- [Multi-Agent Constrained Policy Optimisation](https://mlsanity.com/paper/2110.02793/) (2021)
- [Feudal Multi-Agent Hierarchies for Cooperative Reinforcement Learning](https://mlsanity.com/paper/1901.08492/) (2019)
- [Multi-Agent Reinforcement Learning for Autonomous Driving: A Survey](https://mlsanity.com/paper/2408.09675/) (2024)
- [Solving Stabilize-Avoid Optimal Control via Epigraph Form and Deep Reinforcement Learning](https://mlsanity.com/paper/2305.14154/) (2023)
- [Safe Optimal Control Using Stochastic Barrier Functions and Deep Forward-Backward SDEs](https://mlsanity.com/paper/2009.01196/) (2020)
- [GCBF+: A Neural Graph Control Barrier Function Framework for Distributed Safe Multi-Agent Control](https://mlsanity.com/paper/2401.14554/) (2024)
- [A predictive safety filter for learning-based racing control](https://mlsanity.com/paper/2102.11907/) (2021)
- [Decentralized Control of Quadrotor Swarms with End-to-end Deep Reinforcement Learning](https://mlsanity.com/paper/2109.07735/) (2021)
- [HAVEN: Hierarchical Cooperative Multi-Agent Reinforcement Learning with Dual Coordination Mechanism](https://mlsanity.com/paper/2110.07246/) (2021)
- [Safe Reinforcement Learning of Dynamic High-Dimensional Robotic Tasks: Navigation, Manipulation, Interaction](https://mlsanity.com/paper/2209.13308/) (2022)
- [AutoCost: Evolving Intrinsic Cost for Zero-violation Reinforcement Learning](https://mlsanity.com/paper/2301.10339/) (2023)

## Abstract (arXiv)

Multi-agent systems are widely used in safety-critical applications that require coordinated behavior under strict safety constraints. Existing approaches face a fundamental trade-off: learning-based methods achieve strong empirical performance but lack theoretical safety guarantees, while control-theoretic methods enforce safety but often lead to overly conservative and inefficient behaviors. We propose a hierarchical multi-agent reinforcement learning framework that enforces hard safety constraints under mild assumptions at low level via a constraint manifold, while enabling effective coordination through high-level policy learning. Our approach provides theoretical safety guarantees in the multi-agent setting and yields stationary learning dynamics, thereby enabling stable and efficient training. Empirically, our method achieves competitive performance while maintaining nearly perfect safety rates, and generalizes effectively to varying numbers of agents and obstacles.