# Gossip-based Actor-Learner Architectures for Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/1906.04585 · HTML: https://mlsanity.com/paper/1906.04585/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Jun 2019.

## Built on by (most cited first, 16 of 16)

- [The Blessing of Heterogeneity in Federated Q-Learning: Linear Speedup and Beyond](https://mlsanity.com/paper/2305.10697/) (2023)
- [Towards a multi-stakeholder value-based assessment framework for algorithmic systems](https://mlsanity.com/paper/2205.04525/) (2022)
- [Communication Efficient Parallel Reinforcement Learning](https://mlsanity.com/paper/2102.10740/) (2021)
- [The Sample-Communication Complexity Trade-off in Federated Q-Learning](https://mlsanity.com/paper/2408.16981/) (2024)
- [Learn to Code Sustainably: An Empirical Study on LLM-based Green Code Generation](https://mlsanity.com/paper/2403.03344/) (2024)
- [Compressed Federated Reinforcement Learning with a Generative Model](https://mlsanity.com/paper/2404.10635/) (2024)
- [Gap-Dependent Bounds for Federated Q-learning](https://mlsanity.com/paper/2502.02859/) (2025)
- [Revisiting Gossip Protocols: A Vision for Emergent Coordination in Agentic Multi-Agent Systems](https://mlsanity.com/paper/2508.01531/) (2025)
- [Review of Metrics to Measure the Stability, Robustness and Resilience of Reinforcement Learning](https://mlsanity.com/paper/2203.12048/) (2022)
- [Collaborative Value Function Estimation Under Model Mismatch: A Federated Temporal Difference Analysis](https://mlsanity.com/paper/2503.17454/) (2025)
- [Regret-Optimal Q-Learning with Low Cost for Single-Agent and Federated Reinforcement Learning](https://mlsanity.com/paper/2506.04626/) (2025)
- [Fully Distributed Actor-Critic Architecture for Multitask Deep Reinforcement Learning](https://mlsanity.com/paper/2110.12306/) (2021)
- [Efficient Distributed DNNs in the Mobile-edge-cloud Continuum](https://mlsanity.com/paper/2202.11349/) (2022)
- [Highly Parallelized Reinforcement Learning Training with Relaxed Assignment Dependencies](https://mlsanity.com/paper/2502.20190/) (2025)
- [Pay Attention to What and Where? Interpretable Feature Extractor in Vision-based Deep Reinforcement Learning](https://mlsanity.com/paper/2504.10071/) (2025)
- [From Federated Learning to X-Learning: Breaking the Barriers of Decentrality Through Random Walks](https://mlsanity.com/paper/2509.03709/) (2025)

## Abstract (arXiv)

Multi-simulator training has contributed to the recent success of Deep Reinforcement Learning by stabilizing learning and allowing for higher training throughputs. We propose Gossip-based Actor-Learner Architectures (GALA) where several actor-learners (such as A2C agents) are organized in a peer-to-peer communication topology, and exchange information through asynchronous gossip in order to take advantage of a large number of distributed simulators. We prove that GALA agents remain within an epsilon-ball of one-another during training when using loosely coupled asynchronous communication. By reducing the amount of synchronization between agents, GALA is more computationally efficient and scalable compared to A2C, its fully-synchronous counterpart. GALA also outperforms A2C, being more robust and sample efficient. We show that we can run several loosely coupled GALA agents in parallel on a single GPU and achieve significantly higher hardware utilization and frame-rates than vanilla A2C at comparable power draws.