# Merging model-based control with multi-agent reinforcement learning for multi-agent cooperative teaming strategies

arXiv: https://arxiv.org/abs/2606.06011 · HTML: https://mlsanity.com/paper/2606.06011/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 36 references that are in the map.
- First posted on arXiv 4 Jun 2026.

## Builds on (most cited first, 22 of 22)

- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games](https://mlsanity.com/paper/2103.01955/) (2021)
- [The StarCraft Multi-Agent Challenge](https://mlsanity.com/paper/1902.04043/) (2019)
- [Learning to Communicate with Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1605.06676/) (2016)
- [Learning Multiagent Communication with Backpropagation](https://mlsanity.com/paper/1605.07736/) (2016)
- [Differentiable MPC for End-to-end Planning and Control](https://mlsanity.com/paper/1810.13400/) (2018)
- [acados: a modular open-source framework for fast embedded optimal control](https://mlsanity.com/paper/1910.13753/) (2019)
- [Pontryagin Differentiable Programming: An End-to-End Learning and Control Framework](https://mlsanity.com/paper/1912.12970/) (2019)
- [An Introduction to Centralized Training for Decentralized Execution in Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2409.03052/) (2024)
- [Low Level Control of a Quadrotor with Deep Model-Based Reinforcement Learning](https://mlsanity.com/paper/1901.03737/) (2019)
- [Policy Search for Model Predictive Control with Application to Agile Drone Flight](https://mlsanity.com/paper/2112.03850/) (2021)
- [Actor-Critic Model Predictive Control: Differentiable Optimization meets Reinforcement Learning for Agile Flight](https://mlsanity.com/paper/2306.09852/) (2023)
- [Differentiable Model Predictive Control on the GPU](https://mlsanity.com/paper/2510.06179/) (2025)
- [Predictive Control with Learning-Based Terminal Costs Using Approximate Value Iteration](https://mlsanity.com/paper/2212.00361/) (2022)
- [Imitation Learning from Nonlinear MPC via the Exact Q-Loss and its Gauss-Newton Approximation](https://mlsanity.com/paper/2304.01782/) (2023)
- [Differentiable Robust Model Predictive Control](https://mlsanity.com/paper/2308.08426/) (2023)
- [Fully Decentralized Cooperative Multi-Agent Reinforcement Learning: A Survey](https://mlsanity.com/paper/2401.04934/) (2024)
- [Fast integrators with sensitivity propagation for use in CasADi](https://mlsanity.com/paper/2211.01982/) (2022)
- [Differentiable Optimal Control via Differential Dynamic Programming](https://mlsanity.com/paper/2209.01117/) (2022)
- [Multi-Agent Reinforcement Learning via Distributed MPC as a Function Approximator](https://mlsanity.com/paper/2312.05166/) (2023)
- [PlanNetX: Learning an Efficient Neural Network Planner from MPC for Longitudinal Control](https://mlsanity.com/paper/2404.18863/) (2024)
- [Crazyflow: An Accurate, GPU-Accelerated, Differentiable Drone Simulator in JAX](https://mlsanity.com/paper/2606.01478/) (2026)

## Abstract (arXiv)

In this work, we propose a framework that combines multi-agent reinforcement learning (MARL) with model-based control to achieve safe, dynamically feasible actions in cooperative multi-agent tasks. Multi-agent reinforcement learning provides the advantage of learning cooperative policies for multi-agent teams from discrete non-differentiable rewards in a long planning horizon. Model-predictive control is robust and offers safe, dynamically feasible actions in a fast replanning framework for short horizons. We propose an algorithm that extends actor-critic model predictive control for MARL which we refer to as multi-agent actor-critic model predictive control (MA-AC-MPC). We demonstrate the capabilities of this algorithm by applying it to a multi-agent pursuit-evasion scenario. Specifically, we compare the evader team's strategy using the MA-AC-MPC model and a multi-layer perceptron model (MA-AC-MLP). The pursuer team uses augmented proportional navigation as it is accepted as an advanced adversarial control law. We also provide an example with a heterogeneous environment where a drone and omni-wheeled rover cooperate to achieve repeatable and successful landing with 100% success rate in hardware for MA-AC-MPC compared to 60% for MA-AC-MLP. We demonstrate the robustness of the proposed MA-AC-MPC algorithm in hardware for both environments.