# Distributed Cooperative Multi-Agent Reinforcement Learning with Directed Coordination Graph

arXiv: https://arxiv.org/abs/2201.04962 · HTML: https://mlsanity.com/paper/2201.04962/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 14 references that are in the map.
- First posted on arXiv 10 Jan 2022.

## Builds on (most cited first, 5 of 5)

- [Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agents](https://mlsanity.com/paper/1802.08757/) (2018)
- [Contrasting Exploration in Parameter and Action Space: A Zeroth-Order Optimization Perspective](https://mlsanity.com/paper/1901.11503/) (2019)
- [Multi-Agent Fully Decentralized Value Function Learning with Linear Convergence Rates](https://mlsanity.com/paper/1810.07792/) (2018)
- [Zeroth-order Deterministic Policy Gradient](https://mlsanity.com/paper/2006.07314/) (2020)
- [Cooperative Multi-Agent Reinforcement Learning with Partial Observations](https://mlsanity.com/paper/2006.10822/) (2020)

## Built on by (most cited first, 2 of 2)

- [Efficient Reinforcement Learning for Global Decision Making in the Presence of Local Agents at Scale](https://mlsanity.com/paper/2403.00222/) (2024)
- [Distributed Multi-Agent Reinforcement Learning Based on Graph-Induced Local Value Functions](https://mlsanity.com/paper/2202.13046/) (2022)

## Abstract (arXiv)

Existing distributed cooperative multi-agent reinforcement learning (MARL) frameworks usually assume undirected coordination graphs and communication graphs while estimating a global reward via consensus algorithms for policy evaluation. Such a framework may induce expensive communication costs and exhibit poor scalability due to requirement of global consensus. In this work, we study MARLs with directed coordination graphs, and propose a distributed RL algorithm where the local policy evaluations are based on local value functions. The local value function of each agent is obtained by local communication with its neighbors through a directed learning-induced communication graph, without using any consensus algorithm. A zeroth-order optimization (ZOO) approach based on parameter perturbation is employed to achieve gradient estimation. By comparing with existing ZOO-based RL algorithms, we show that our proposed distributed RL algorithm guarantees high scalability. A distributed resource allocation example is shown to illustrate the effectiveness of our algorithm.