# Graph Policy Gradients for Large Scale Robot Control

arXiv: https://arxiv.org/abs/1907.03822 · HTML: https://mlsanity.com/paper/1907.03822/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Jul 2019.

## Built on by (most cited first, 28 of 28)

- [Scalable Multi-Agent Reinforcement Learning through Intelligent Information Aggregation](https://mlsanity.com/paper/2211.02127/) (2022)
- [On the Global Optimum Convergence of Momentum-based Policy Gradient](https://mlsanity.com/paper/2110.10116/) (2021)
- [Asynchronous Actor-Critic for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2209.10113/) (2022)
- [Graph Neural Networks for Decentralized Multi-Robot Submodular Action Selection](https://mlsanity.com/paper/2105.08601/) (2021)
- [Neurosymbolic Transformers for Multi-Agent Communication](https://mlsanity.com/paper/2101.03238/) (2021)
- [Synthesizing Decentralized Controllers with Graph Neural Networks and Imitation Learning](https://mlsanity.com/paper/2012.14906/) (2020)
- [Beyond Robustness: A Taxonomy of Approaches towards Resilient Multi-Robot Systems](https://mlsanity.com/paper/2109.12343/) (2021)
- [Distributed neural network control with dependability guarantees: a compositional port-Hamiltonian approach](https://mlsanity.com/paper/2112.09046/) (2021)
- [A Framework for Real-World Multi-Robot Systems Running Decentralized GNN-Based Policies](https://mlsanity.com/paper/2111.01777/) (2021)
- [ModGNN: Expert Policy Approximation in Multi-Agent Systems with a Modular Graph Neural Network Architecture](https://mlsanity.com/paper/2103.13446/) (2021)
- [Learning Control Admissibility Models with Graph Neural Networks for Multi-Agent Navigation](https://mlsanity.com/paper/2210.09378/) (2022)
- [Communication Topology Co-Design in Graph Recurrent Neural Network Based Distributed Control](https://mlsanity.com/paper/2104.13868/) (2021)
- [Large Scale Distributed Collaborative Unlabeled Motion Planning with Graph Policy Gradients](https://mlsanity.com/paper/2102.06284/) (2021)
- [Learning-based Motion Planning in Dynamic Environments Using GNNs and Temporal Encoding](https://mlsanity.com/paper/2210.08408/) (2022)
- [Graph learning in robotics: a survey](https://mlsanity.com/paper/2310.04294/) (2023)
- [Cooperation and Fairness in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2410.14916/) (2024)
- [The Holy Grail of Multi-Robot Planning: Learning to Generate Online-Scalable Solutions from Offline-Optimal Experts](https://mlsanity.com/paper/2107.12254/) (2021)
- [Capability-Aware Shared Hypernetworks for Flexible Heterogeneous Multi-Robot Coordination](https://mlsanity.com/paper/2501.06058/) (2025)
- [Counterfactual Explanation Policies in RL](https://mlsanity.com/paper/2307.13192/) (2023)
- [Generalization of Heterogeneous Multi-Robot Policies via Awareness and Communication of Capabilities](https://mlsanity.com/paper/2401.13127/) (2024)
- [Graph Neural Networks Gone Hogwild](https://mlsanity.com/paper/2407.00494/) (2024)
- [Hierarchical Message-Passing Policies for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2507.23604/) (2025)
- [TERL: Large-Scale Multi-Target Encirclement Using Transformer-Enhanced Reinforcement Learning](https://mlsanity.com/paper/2503.12395/) (2025)
- [Foresight of Graph Reinforcement Learning Latent Permutations Learnt by Gumbel Sinkhorn Network](https://mlsanity.com/paper/2110.12144/) (2021)
- [An Advantage-based Optimization Method for Reinforcement Learning in Large Action Space](https://mlsanity.com/paper/2412.12605/) (2024)
- [Beyond Fixed Morphologies: Learning Graph Policies with Trust Region Compensation in Variable Action Spaces](https://mlsanity.com/paper/2508.14102/) (2025)
- [Network Distributed Multi-Agent Reinforcement Learning for Consensus Control of Quadcopters](https://mlsanity.com/paper/2606.02107/) (2026)
- [Action-Factored Multi-Agent Reinforcement Learning for Scalable Quantum Device Tuning](https://mlsanity.com/paper/2607.09422/) (2026)

## Abstract (arXiv)

In this paper, we consider the problem of learning policies to control a large number of homogeneous robots. To this end, we propose a new algorithm we call Graph Policy Gradients (GPG) that exploits the underlying graph symmetry among the robots. The curse of dimensionality one encounters when working with a large number of robots is mitigated by employing a graph convolutional neural (GCN) network to parametrize policies for the robots. The GCN reduces the dimensionality of the problem by learning filters that aggregate information among robots locally, similar to how a convolutional neural network is able to learn local features in an image. Through experiments on formation flying, we show that our proposed method is able to scale better than existing reinforcement methods that employ fully connected networks. More importantly, we show that by using our locally learned filters we are able to zero-shot transfer policies trained on just three robots to over hundred robots.