mlsanityData to 7 Oct 2026

Communication Efficient Parallel Reinforcement Learning

arXiv 2102.10740 ↗First posted 22 Feb 2021cs.LG cs.AI cs.MAcited by 10

Not on the map. Why?

Abstract (arXiv)

We consider the problem where M agents interact with M identical and independent environments with S states and A actions using reinforcement learning for T rounds. The agents share their data with a central server to minimize their regret. We aim to find an algorithm that allows the agents to minimize the regret with infrequent communication rounds. We provide NAM which runs at each agent and prove that the total cumulative regret of M agents is upper bounded as TildeO(DS√MAT) for a Markov Decision Process with diameter D, number of states S, and number of actions A. The agents synchronize after their visitations to any state-action pair exceeds a certain threshold. Using this, we obtain a bound of Oleft(MSAlog(MT)right) on the total number of communications rounds. Finally, we evaluate the algorithm against multiple environments and demonstrate that the proposed algorithm performs at par with an always communication version of the UCRL2 algorithm, while with significantly lower communication.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 21 of its 41 references that are in the map. First posted on arXiv 22 Feb 2021.

Builds on

21 of 41 references.

References · 21

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  4. 2015Trust Region Policy Optimizationcited by 2,503
  5. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  6. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  7. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 698
  8. 2018Is Q-learning Provably Efficient?cited by 485
  9. 2018Distributed Prioritized Experience Replaycited by 260
  10. 2013(More) Efficient Reinforcement Learning via Posterior Samplingcited by 233
  11. 2018Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agentscited by 189
  12. 2020Differentially-Private Federated Linear Banditscited by 83
9 more
  1. 2015Massively Parallel Methods for Deep Reinforcement Learningcited by 80
  2. 2019DeepPool: Distributed Model-free Algorithm for Ride-sharing using Deep Reinforcement Learningcited by 54
  3. 2019Social Learning in Multi Agent Multi Armed Banditscited by 48
  4. 2013Distributed Exploration in Multi-Armed Banditscited by 46
  5. 2020The Gossiping Insert-Eliminate Algorithm for Multi-Agent Banditscited by 37
  6. 2017Efficient Parallel Methods for Deep Reinforcement Learningcited by 32
  7. 2019Gossip-based Actor-Learner Architectures for Deep Reinforcement Learningcited by 16
  8. 2019Near-optimal Optimistic Reinforcement Learning using Empirical Bernstein Inequalitiescited by 15
  9. 2020FlexPool: A Distributed Model-Free Deep Reinforcement Learning Algorithm for Joint Passengers & Goods Transportationcited by 6

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 2; 2023: 2; 2024: 2; 2025: 2.

10 papers.

  1. 2021Distributed Learning in Wireless Networks: Recent Progress and Future Challengescited by 125
  2. 2024Asynchronous Federated Reinforcement Learning with Policy Gradient Updates: Algorithm Design and Convergence Analysiscited by 21
  3. 2023Improved Communication Efficiency in Federated Natural Policy Gradient via ADMM-based Gradient Updatescited by 20
  4. 2023Multi-task Hierarchical Adversarial Inverse Reinforcement Learningcited by 15
  5. 2022Byzantine-Robust Online and Offline Distributed Reinforcement Learningcited by 13
  6. 2024Global Convergence Guarantees for Federated Policy Gradient Methods with Adversariescited by 6
  7. 2025Gap-Dependent Bounds for Federated Q-learningcited by 4
  8. 2022One Policy is Enough: Parallel Exploration with a Single Policy is Near-Optimal for Reward-Free Reinforcement Learningcited by 2
  9. 2025Regret-Optimal Q-Learning with Low Cost for Single-Agent and Federated Reinforcement Learningcited by 2
  10. 2021Federated Reinforcement Learning at the Edgecited by 1