# Communication-Efficient Policy Gradient Methods for Distributed Reinforcement Learning

arXiv: https://arxiv.org/abs/1812.03239 · HTML: https://mlsanity.com/paper/1812.03239/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Dec 2018.

## Built on by (most cited first, 11 of 11)

- [FedKL: Tackling Data Heterogeneity in Federated Reinforcement Learning by Penalizing KL Divergence](https://mlsanity.com/paper/2204.08125/) (2022)
- [Pervasive AI for IoT applications: A Survey on Resource-efficient Distributed Artificial Intelligence](https://mlsanity.com/paper/2105.01798/) (2021)
- [Federated Offline Reinforcement Learning](https://mlsanity.com/paper/2206.05581/) (2022)
- [MARL with General Utilities via Decentralized Shadow Reward Actor-Critic](https://mlsanity.com/paper/2106.00543/) (2021)
- [Network Support for High-performance Distributed Machine Learning](https://mlsanity.com/paper/2102.03394/) (2021)
- [Sample and Communication Efficient Fully Decentralized MARL Policy Evaluation via a New Approach: Local TD update](https://mlsanity.com/paper/2403.15935/) (2024)
- [Linear Regression over Networks with Communication Guarantees](https://mlsanity.com/paper/2103.04140/) (2021)
- [AI in Human-computer Gaming: Techniques, Challenges and Opportunities](https://mlsanity.com/paper/2111.07631/) (2021)
- [Adaptive Stochastic ADMM for Decentralized Reinforcement Learning in Edge Industrial IoT](https://mlsanity.com/paper/2107.00481/) (2021)
- [Event-Based Communication in Distributed Q-Learning](https://mlsanity.com/paper/2109.01417/) (2021)
- [Robust Event-Driven Interactions in Cooperative Multi-Agent Learning](https://mlsanity.com/paper/2204.03361/) (2022)

## Abstract (arXiv)

This paper deals with distributed policy optimization in reinforcement learning, which involves a central controller and a group of learners. In particular, two typical settings encountered in several applications are considered: multi-agent reinforcement learning (RL) and parallel RL, where frequent information exchanges between the learners and the controller are required. For many practical distributed systems, however, the overhead caused by these frequent communication exchanges is considerable, and becomes the bottleneck of the overall performance. To address this challenge, a novel policy gradient approach is developed for solving distributed RL. The novel approach adaptively skips the policy gradient communication during iterations, and can reduce the communication overhead without degrading learning performance. It is established analytically that: i) the novel algorithm has convergence rate identical to that of the plain-vanilla policy gradient; while ii) if the distributed learners are heterogeneous in terms of their reward functions, the number of communication rounds needed to achieve a desirable learning accuracy is markedly reduced. Numerical experiments corroborate the communication reduction attained by the novel algorithm compared to alternatives.