# Learning Decentralized Partially Observable Mean Field Control for Artificial Collective Behavior

arXiv: https://arxiv.org/abs/2307.06175 · HTML: https://mlsanity.com/paper/2307.06175/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 83 references that are in the map.
- First posted on arXiv 12 Jul 2023.

## Builds on (most cited first, 30 of 37)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games](https://mlsanity.com/paper/2103.01955/) (2021)
- [Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms](https://mlsanity.com/paper/1911.10635/) (2019)
- [The Complexity of Decentralized Control of Markov Decision Processes](https://mlsanity.com/paper/1301.3836/) (2013)
- [Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks](https://mlsanity.com/paper/2006.07869/) (2020)
- [Mean Field Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1802.05438/) (2018)
- [Deep Reinforcement Learning for Intelligent Transportation Systems: A Survey](https://mlsanity.com/paper/2005.00935/) (2020)
- [MAgent: A Many-Agent Reinforcement Learning Platform for Artificial Collective Intelligence](https://mlsanity.com/paper/1712.00600/) (2017)
- [Fictitious Play for Mean Field Games: Continuous Time Analysis and Applications](https://mlsanity.com/paper/2007.03458/) (2020)
- [A Unified Algebraic Perspective on Lipschitz Neural Networks](https://mlsanity.com/paper/2303.03169/) (2023)
- [Model-Free Mean-Field Reinforcement Learning: Mean-Field MDP and Mean-Field Q-Learning](https://mlsanity.com/paper/1910.12802/) (2019)
- [Mean-Field Controls with Q-learning for Cooperative MARL: Convergence and Complexity Analysis](https://mlsanity.com/paper/2002.04131/) (2020)
- [Efficient Model-Based Multi-Agent Mean-Field Reinforcement Learning](https://mlsanity.com/paper/2107.04050/) (2021)
- [On the Approximation of Cooperative Heterogeneous Multi-Agent Reinforcement Learning (MARL) using Mean Field Control (MFC)](https://mlsanity.com/paper/2109.04024/) (2021)
- [Mean Field Games Flock! The Reinforcement Learning Way](https://mlsanity.com/paper/2105.07933/) (2021)
- [Linear-Quadratic Mean-Field Reinforcement Learning: Convergence of Policy Gradient Methods](https://mlsanity.com/paper/1910.04295/) (2019)
- [Dynamic Programming Principles for Mean-Field Controls with Learning](https://mlsanity.com/paper/1911.07314/) (2019)
- [Decentralized Multi-Agent Reinforcement Learning with Networked Agents: Recent Advances](https://mlsanity.com/paper/1912.03821/) (2019)
- [Global Convergence of Policy Gradient for Linear-Quadratic Mean-Field Control/Game in Continuous Time](https://mlsanity.com/paper/2008.06845/) (2020)
- [Actor-Critic Provably Finds Nash Equilibria of Linear-Quadratic Mean-Field Games](https://mlsanity.com/paper/1910.07498/) (2019)
- [Partially Observable Mean Field Reinforcement Learning](https://mlsanity.com/paper/2012.15791/) (2020)
- [On the Statistical Efficiency of Mean-Field Reinforcement Learning with General Function Approximation](https://mlsanity.com/paper/2305.11283/) (2023)
- [Decentralized Mean Field Games](https://mlsanity.com/paper/2112.09099/) (2021)
- [Bellman equation and viscosity solutions for mean-field stochastic control problem](https://mlsanity.com/paper/1512.07866/) (2015)
- [Mean-Field-Type Games in Engineering](https://mlsanity.com/paper/1605.03281/) (2016)
- [Reinforcement learning of optimal active particle navigation](https://mlsanity.com/paper/2202.00812/) (2022)
- [MFGLib: A Library for Mean-Field Games](https://mlsanity.com/paper/2304.08630/) (2023)

## Built on by (most cited first, 7 of 7)

- [Reinforcement Learning for Finite Space Mean-Field Type Games](https://mlsanity.com/paper/2409.18152/) (2024)
- [Collaborative Optimization of the Age of Information under Partial Observability](https://mlsanity.com/paper/2312.12977/) (2023)
- [Learning to Stop: Deep Learning for Mean Field Optimal Stopping](https://mlsanity.com/paper/2410.08850/) (2024)
- [Fully-Decentralized MADDPG with Networked Agents](https://mlsanity.com/paper/2503.06747/) (2025)
- [Vulnerable Agent Identification in Large-Scale Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2509.15103/) (2025)
- [Mean-Field Control on Sparse Graphs: From Local Limits to GNNs via Neighborhood Distributions](https://mlsanity.com/paper/2601.21477/) (2026)
- [Beyond Nominal Equilibria: Risk-Averse Multi-Population Mean-Field Games](https://mlsanity.com/paper/2610.09244/) (2026)

## Abstract (arXiv)

Recent reinforcement learning (RL) methods have achieved success in various domains. However, multi-agent RL (MARL) remains a challenge in terms of decentralization, partial observability and scalability to many agents. Meanwhile, collective behavior requires resolution of the aforementioned challenges, and remains of importance to many state-of-the-art applications such as active matter physics, self-organizing systems, opinion dynamics, and biological or robotic swarms. Here, MARL via mean field control (MFC) offers a potential solution to scalability, but fails to consider decentralized and partially observable systems. In this paper, we enable decentralized behavior of agents under partial information by proposing novel models for decentralized partially observable MFC (Dec-POMFC), a broad class of problems with permutation-invariant agents allowing for reduction to tractable single-agent Markov decision processes (MDP) with single-agent RL solution. We provide rigorous theoretical results, including a dynamic programming principle, together with optimality guarantees for Dec-POMFC solutions applied to finite swarms of interest. Algorithmically, we propose Dec-POMFC-based policy gradient methods for MARL via centralized training and decentralized execution, together with policy gradient approximation guarantees. In addition, we improve upon state-of-the-art histogram-based MFC by kernel methods, which is of separate interest also for fully observable MFC. We evaluate numerically on representative collective behavior tasks such as adapted Kuramoto and Vicsek swarming models, being on par with state-of-the-art MARL. Overall, our framework takes a step towards RL-based engineering of artificial collective behavior via MFC.