# Hierarchical Reinforcement Learning with Low-Level MPC for Multi-Agent Control

arXiv: https://arxiv.org/abs/2509.15799 · HTML: https://mlsanity.com/paper/2509.15799/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 26 references that are in the map.
- First posted on arXiv 19 Sep 2025.

## Builds on (most cited first, 15 of 15)

- [Safe Model-based Reinforcement Learning with Stability Guarantees](https://mlsanity.com/paper/1705.08551/) (2017)
- [A predictive safety filter for learning-based control of constrained nonlinear dynamical systems](https://mlsanity.com/paper/1812.05506/) (2018)
- [Safe Reinforcement Learning Using Robust MPC](https://mlsanity.com/paper/1906.04005/) (2019)
- [State-wise Safe Reinforcement Learning: A Survey](https://mlsanity.com/paper/2302.03122/) (2023)
- [Robust Model Predictive Shielding for Safe Reinforcement Learning with Stochastic Dynamics](https://mlsanity.com/paper/1910.10885/) (2019)
- [Synthesis of Model Predictive Control and Reinforcement Learning: Survey and Classification](https://mlsanity.com/paper/2502.02133/) (2025)
- [MARLlib: A Scalable and Efficient Multi-agent Reinforcement Learning Library](https://mlsanity.com/paper/2210.13708/) (2022)
- [Learning-based Model Predictive Control for Safe Exploration and Reinforcement Learning](https://mlsanity.com/paper/1906.12189/) (2019)
- [Learning High-Level Policies for Model Predictive Control](https://mlsanity.com/paper/2007.10284/) (2020)
- [A Hierarchical Approach for Strategic Motion Planning in Autonomous Racing](https://mlsanity.com/paper/2212.01607/) (2022)
- [Safe Multi-Agent Reinforcement Learning for Behavior-Based Cooperative Navigation](https://mlsanity.com/paper/2312.12861/) (2023)
- [Safety Guaranteed Robust Multi-Agent Reinforcement Learning with Hierarchical Control for Connected and Automated Vehicles](https://mlsanity.com/paper/2309.11057/) (2023)
- [On feasibility, stability and performance in distributed model predictive control](https://mlsanity.com/paper/1302.1974/) (2013)
- [DeepSafeMPC: Deep Learning-Based Model Predictive Control for Safe Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2403.06397/) (2024)
- [A Provable Approach for End-to-End Safe Reinforcement Learning](https://mlsanity.com/paper/2505.21852/) (2025)

## Abstract (arXiv)

Achieving safe and coordinated behavior in dynamic, constraint-rich environments remains a major challenge for learning-based control. Pure end-to-end learning often suffers from poor sample efficiency and limited reliability, while model-based methods depend on predefined references and struggle to generalize. We propose a hierarchical framework that combines tactical decision-making via reinforcement learning (RL) with low-level execution through Model Predictive Control (MPC). For the case of multi-agent systems this means that high-level policies select abstract targets from structured regions of interest (ROIs), while MPC ensures dynamically feasible and safe motion. Tested on a predator-prey benchmark, our approach outperforms end-to-end and shielding-based RL baselines in terms of reward, safety, and consistency, underscoring the benefits of combining structured learning with model-based control.