# Distributed Multi-Agent Reinforcement Learning Based on Graph-Induced Local Value Functions

arXiv: https://arxiv.org/abs/2202.13046 · HTML: https://mlsanity.com/paper/2202.13046/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 60 references that are in the map.
- First posted on arXiv 26 Feb 2022.

## Builds on (most cited first, 24 of 24)

- [Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms](https://mlsanity.com/paper/1911.10635/) (2019)
- [Value-Decomposition Networks For Cooperative Multi-Agent Learning](https://mlsanity.com/paper/1706.05296/) (2017)
- [Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulator](https://mlsanity.com/paper/1801.05039/) (2018)
- [Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agents](https://mlsanity.com/paper/1802.08757/) (2018)
- [Global Optimality Guarantees For Policy Gradient Methods](https://mlsanity.com/paper/1906.01786/) (2019)
- [Deep Decentralized Multi-task Multi-Agent Reinforcement Learning under Partial Observability](https://mlsanity.com/paper/1703.06182/) (2017)
- [A Review of Cooperative Multi-Agent Deep Reinforcement Learning](https://mlsanity.com/paper/1908.03963/) (2019)
- [Derivative-Free Methods for Policy Optimization: Guarantees for Linear Quadratic Systems](https://mlsanity.com/paper/1812.08305/) (2018)
- [Scalable Multi-Agent Reinforcement Learning for Networked Systems with Average Reward](https://mlsanity.com/paper/2006.06626/) (2020)
- [Distributed Reinforcement Learning for Decentralized Linear Quadratic Control: A Derivative-Free Policy Optimization Approach](https://mlsanity.com/paper/1912.09135/) (2019)
- [Multi-Agent Reinforcement Learning in Stochastic Networked Systems](https://mlsanity.com/paper/2006.06555/) (2020)
- [Contrasting Exploration in Parameter and Action Space: A Zeroth-Order Optimization Perspective](https://mlsanity.com/paper/1901.11503/) (2019)
- [A New One-Point Residual-Feedback Oracle For Black-Box Learning and Control](https://mlsanity.com/paper/2006.10820/) (2020)
- [Zeroth Order Nonconvex Multi-Agent Optimization over Networks](https://mlsanity.com/paper/1710.09997/) (2017)
- [Zeroth-order Deterministic Policy Gradient](https://mlsanity.com/paper/2006.07314/) (2020)
- [Distributed Zero-Order Optimization under Adversarial Noise](https://mlsanity.com/paper/2102.01121/) (2021)
- [Cooperative Multi-Agent Reinforcement Learning with Partial Observations](https://mlsanity.com/paper/2006.10822/) (2020)
- [Diff-DAC: Distributed Actor-Critic for Average Multitask Deep Reinforcement Learning](https://mlsanity.com/paper/1710.10363/) (2017)
- [Distributed Zero-Order Algorithms for Nonconvex Multi-Agent Optimization](https://mlsanity.com/paper/1908.11444/) (2019)
- [Model-Free Optimal Control of Linear Multi-Agent Systems via Decomposition and Hierarchical Approximation](https://mlsanity.com/paper/2008.06604/) (2020)
- [Asynchronous Distributed Reinforcement Learning for LQR Control via Zeroth-Order Block Coordinate Descent](https://mlsanity.com/paper/2107.12416/) (2021)
- [Linear Time Average Consensus on Fixed Graphs and Implications for Decentralized Optimization and Multi-Agent Control](https://mlsanity.com/paper/1411.4186/) (2014)
- [Privacy-Preserving Distributed Zeroth-Order Optimization](https://mlsanity.com/paper/2008.13468/) (2020)
- [Distributed Cooperative Multi-Agent Reinforcement Learning with Directed Coordination Graph](https://mlsanity.com/paper/2201.04962/) (2022)

## Built on by (most cited first, 6 of 6)

- [Exploiting inter-agent coupling information for efficient reinforcement learning of cooperative LQR](https://mlsanity.com/paper/2504.20927/) (2025)
- [Factored Value Functions for Graph-Based Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2601.11401/) (2026)
- [Action Dependency Graphs for Globally Optimal Coordinated Reinforcement Learning](https://mlsanity.com/paper/2506.00797/) (2025)
- [Structured Cooperative Multi-Agent Reinforcement Learning: a Bayesian Network Perspective](https://mlsanity.com/paper/2510.09937/) (2025)
- [Homomorphic Mappings for Value-Preserving State Aggregation in Markov Decision Processes](https://mlsanity.com/paper/2510.09965/) (2025)
- [Explicit Credit Assignment through Local Rewards and Dependence Graphs in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2601.21523/) (2026)

## Abstract (arXiv)

Achieving distributed reinforcement learning (RL) for large-scale cooperative multi-agent systems (MASs) is challenging because: (i) each agent has access to only limited information; (ii) issues on convergence or computational complexity emerge due to the curse of dimensionality. In this paper, we propose a general computationally efficient distributed framework for cooperative multi-agent reinforcement learning (MARL) by utilizing the structures of graphs involved in this problem. We introduce three coupling graphs describing three types of inter-agent couplings in MARL, namely, the state graph, the observation graph and the reward graph. By further considering a communication graph, we propose two distributed RL approaches based on local value-functions derived from the coupling graphs. The first approach is able to reduce sample complexity significantly under specific conditions on the aforementioned four graphs. The second approach provides an approximate solution and can be efficient even for problems with dense coupling graphs. Here there is a trade-off between minimizing the approximation error and reducing the computational complexity. Simulations show that our RL algorithms have a significantly improved scalability to large-scale MASs compared with centralized and consensus-based distributed RL algorithms.