# Scalable Reinforcement Learning for Multi-Agent Networked Systems

arXiv: https://arxiv.org/abs/1912.02906 · HTML: https://mlsanity.com/paper/1912.02906/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Dec 2019.

## Built on by (most cited first, 24 of 24)

- [Multi-agent Reinforcement Learning: A Comprehensive Survey](https://mlsanity.com/paper/2312.10256/) (2023)
- [Scalable Primal-Dual Actor-Critic Method for Safe Multi-Agent RL with General Utilities](https://mlsanity.com/paper/2305.17568/) (2023)
- [Multi-Player Zero-Sum Markov Games with Networked Separable Interactions](https://mlsanity.com/paper/2307.09470/) (2023)
- [BMG-Q: Localized Bipartite Match Graph Attention Q-Learning for Ride-Pooling Order Dispatch](https://mlsanity.com/paper/2501.13448/) (2025)
- [Locally Interdependent Multi-Agent MDP: Theoretical Framework for Decentralized Agents with Dynamic Dependencies](https://mlsanity.com/paper/2406.06823/) (2024)
- [Quantile-Based Deep Reinforcement Learning using Two-Timescale Policy Gradient Algorithms](https://mlsanity.com/paper/2305.07248/) (2023)
- [Representation Learning For Efficient Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2406.02890/) (2024)
- [Scalable spectral representations for multi-agent reinforcement learning in network MDPs](https://mlsanity.com/paper/2410.17221/) (2024)
- [Building Better Environments for Autonomous Cyber Defence](https://mlsanity.com/paper/2604.08805/) (2026)
- [Scalable and Sample Efficient Distributed Policy Gradient Algorithms in Multi-Agent Networked Systems](https://mlsanity.com/paper/2212.06357/) (2022)
- [Scalable Multi-Agent Reinforcement Learning with General Utilities](https://mlsanity.com/paper/2302.07938/) (2023)
- [A Scalable Network-Aware Multi-Agent Reinforcement Learning Framework for Decentralized Inverter-based Voltage Control](https://mlsanity.com/paper/2312.04371/) (2023)
- [Scalable Reinforcement Learning for Linear-Quadratic Control of Networks](https://mlsanity.com/paper/2401.16183/) (2024)
- [Reinforcement-Learning Portfolio Allocation with Dynamic Embedding of Market Information](https://mlsanity.com/paper/2501.17992/) (2025)
- [Learning Approximate Nash Equilibria in Cooperative Multi-Agent Reinforcement Learning via Mean-Field Subsampling](https://mlsanity.com/paper/2603.03759/) (2026)
- [Peer-to-Peer Energy Trading of Solar and Energy Storage: A Networked Multiagent Reinforcement Learning Approach](https://mlsanity.com/paper/2401.13947/) (2024)
- [A finite time analysis of distributed Q-learning](https://mlsanity.com/paper/2405.14078/) (2024)
- [Multi-Agent Conditional Diffusion Model with Mean Field Communication as Wireless Resource Allocation Planner](https://mlsanity.com/paper/2510.22969/) (2025)
- [Revisiting Policy Gradients for Restricted Policy Classes: Escaping Myopic Local Optima with k-step Policy Gradients](https://mlsanity.com/paper/2605.10909/) (2026)
- [A New Policy Iteration Algorithm For Reinforcement Learning in Zero-Sum Markov Games](https://mlsanity.com/paper/2303.09716/) (2023)
- [Signal attenuation enables scalable decentralized multi-agent reinforcement learning over networks](https://mlsanity.com/paper/2505.11461/) (2025)
- [Structured Cooperative Multi-Agent Reinforcement Learning: a Bayesian Network Perspective](https://mlsanity.com/paper/2510.09937/) (2025)
- [Hybrid Offline-Online Multi-Agent Decision Transformers for Wireless Resource Management](https://mlsanity.com/paper/2608.28878/) (2026)
- [Continual Graph Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2610.10302/) (2026)

## Abstract (arXiv)

We study reinforcement learning (RL) in a setting with a network of agents whose states and actions interact in a local manner where the objective is to find localized policies such that the (discounted) global reward is maximized. A fundamental challenge in this setting is that the state-action space size scales exponentially in the number of agents, rendering the problem intractable for large networks. In this paper, we propose a Scalable Actor Critic (SAC) framework that exploits the network structure and finds a localized policy that is an O(ρkappa)-approximation of a stationary point of the objective for some ρ∈(0,1), with complexity that scales with the local state-action space size of the largest kappa-hop neighborhood of the network. We illustrate our model and approach using examples from wireless communication, epidemics and traffic.