# Policy Gradient With Value Function Approximation For Collective Multiagent Planning

arXiv: https://arxiv.org/abs/1804.02884 · HTML: https://mlsanity.com/paper/1804.02884/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Apr 2018.

## Built on by (most cited first, 6 of 6)

- [A New Formalism, Method and Open Issues for Zero-Shot Coordination](https://mlsanity.com/paper/2106.06613/) (2021)
- [rm E(3)-Equivariant Actor-Critic Methods for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2308.11842/) (2023)
- [Communication-Efficient Actor-Critic Methods for Homogeneous Markov Games](https://mlsanity.com/paper/2202.09422/) (2022)
- [A Survey of Machine Learning-Based Ride-Hailing Planning](https://mlsanity.com/paper/2303.14646/) (2023)
- [A Bibliometric Analysis and Review on Reinforcement Learning for Transportation Applications](https://mlsanity.com/paper/2210.14524/) (2022)
- [Symmetries-enhanced Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2501.01136/) (2025)

## Abstract (arXiv)

Decentralized (PO)MDPs provide an expressive framework for sequential decision making in a multiagent system. Given their computational complexity, recent research has focused on tractable yet practical subclasses of Dec-POMDPs. We address such a subclass called CDEC-POMDP where the collective behavior of a population of agents affects the joint-reward and environment dynamics. Our main contribution is an actor-critic (AC) reinforcement learning method for optimizing CDEC-POMDP policies. Vanilla AC has slow convergence for larger problems. To address this, we show how a particular decomposition of the approximate action-value function over agents leads to effective updates, and also derive a new way to train the critic based on local reward signals. Comparisons on a synthetic benchmark and a real-world taxi fleet optimization problem show that our new AC approach provides better quality solutions than previous best approaches.