# Mean-Field Approximation of Cooperative Constrained Multi-Agent Reinforcement Learning (CMARL)

arXiv: https://arxiv.org/abs/2209.07437 · HTML: https://mlsanity.com/paper/2209.07437/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 40 references that are in the map.
- First posted on arXiv 15 Sep 2022.

## Builds on (most cited first, 23 of 23)

- [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017)
- [QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1803.11485/) (2018)
- [Value-Decomposition Networks For Cooperative Multi-Agent Learning](https://mlsanity.com/paper/1706.05296/) (2017)
- [QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1905.05408/) (2019)
- [Risk-Constrained Reinforcement Learning with Percentile Risk Criteria](https://mlsanity.com/paper/1512.01629/) (2015)
- [Weighted QMIX: Expanding Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2006.10800/) (2020)
- [A General Safety Framework for Learning-Based Control in Uncertain Robotic Systems](https://mlsanity.com/paper/1705.01292/) (2017)
- [Constrained Reinforcement Learning Has Zero Duality Gap](https://mlsanity.com/paper/1910.13393/) (2019)
- [Safe Policies for Reinforcement Learning via Primal-Dual Methods](https://mlsanity.com/paper/1911.09101/) (2019)
- [Accelerated Primal-Dual Policy Optimization for Safe Reinforcement Learning](https://mlsanity.com/paper/1802.06480/) (2018)
- [DeepPool: Distributed Model-free Algorithm for Ride-sharing using Deep Reinforcement Learning](https://mlsanity.com/paper/1903.03882/) (2019)
- [Mean-Field Controls with Q-learning for Cooperative MARL: Convergence and Complexity Analysis](https://mlsanity.com/paper/2002.04131/) (2020)
- [Unified Reinforcement Q-Learning for Mean Field Game and Control Problems](https://mlsanity.com/paper/2006.13912/) (2020)
- [Efficient Model-Based Multi-Agent Mean-Field Reinforcement Learning](https://mlsanity.com/paper/2107.04050/) (2021)
- [Large-Scale Traffic Signal Control Using a Novel Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1908.03761/) (2019)
- [On the Approximation of Cooperative Heterogeneous Multi-Agent Reinforcement Learning (MARL) using Mean Field Control (MFC)](https://mlsanity.com/paper/2109.04024/) (2021)
- [Linear-Quadratic Mean-Field Reinforcement Learning: Convergence of Policy Gradient Methods](https://mlsanity.com/paper/1910.04295/) (2019)
- [Achieving Zero Constraint Violation for Constrained Reinforcement Learning via Conservative Natural Policy Gradient Primal-Dual Algorithm](https://mlsanity.com/paper/2206.05850/) (2022)
- [Optimal resource allocation for competitive spreading processes on bilayer networks](https://mlsanity.com/paper/1512.05299/) (2015)
- [Can Mean Field Control (MFC) Approximate Cooperative Multi Agent Reinforcement Learning (MARL) with Non-Uniform Interaction?](https://mlsanity.com/paper/2203.00035/) (2022)
- [On the Near-Optimality of Local Policies in Large Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2209.03491/) (2022)
- [Joint Latency and Cost Optimization for Erasure-coded Data Center Storage](https://mlsanity.com/paper/1404.4975/) (2014)
- [Iterative Dynamic Water-filling for Fading Multiple-Access Channels with Energy Harvesting](https://mlsanity.com/paper/1401.2376/) (2014)

## Built on by (most cited first, 5 of 5)

- [Scalable Primal-Dual Actor-Critic Method for Safe Multi-Agent RL with General Utilities](https://mlsanity.com/paper/2305.17568/) (2023)
- [Safe Model-Based Multi-Agent Mean-Field Reinforcement Learning](https://mlsanity.com/paper/2306.17052/) (2023)
- [Scalable Ride-Sourcing Vehicle Rebalancing with Service Accessibility Guarantee: A Constrained Mean-Field Reinforcement Learning Approach](https://mlsanity.com/paper/2503.24183/) (2025)
- [Multi-Year Maintenance Planning for Large-Scale Infrastructure Systems: A Novel Network Deep Q-Learning Approach](https://mlsanity.com/paper/2507.18732/) (2025)
- [Hierarchical Deep Reinforcement Learning Framework for Multi-Year Asset Management Under Budget Constraints](https://mlsanity.com/paper/2507.19458/) (2025)

## Abstract (arXiv)

Mean-Field Control (MFC) has recently been proven to be a scalable tool to approximately solve large-scale multi-agent reinforcement learning (MARL) problems. However, these studies are typically limited to unconstrained cumulative reward maximization framework. In this paper, we show that one can use the MFC approach to approximate the MARL problem even in the presence of constraints. Specifically, we prove that, an N-agent constrained MARL problem, with state, and action spaces of each individual agents being of sizes |X|, and |U| respectively, can be approximated by an associated constrained MFC problem with an error, etriangleq Oleft([√|X|+√|U|]/√Nright). In a special case where the reward, cost, and state transition functions are independent of the action distribution of the population, we prove that the error can be improved to e=O(√|X|/√N). Also, we provide a Natural Policy Gradient based algorithm and prove that it can solve the constrained MARL problem within an error of O(e) with a sample complexity of O(e-6).