mlsanityData to 7 Oct 2026

Compressed Federated Reinforcement Learning with a Generative Model

arXiv 2404.10635 ↗First posted 26 Mar 2024cs.DC cs.LG cs.MAcited by 5

Not on the map. Why?

Abstract (arXiv)

Reinforcement learning has recently gained unprecedented popularity, yet it still grapples with sample inefficiency. Addressing this challenge, federated reinforcement learning (FedRL) has emerged, wherein agents collaboratively learn a single policy by aggregating local estimations. However, this aggregation step incurs significant communication costs. In this paper, we propose CompFedRL, a communication-efficient FedRL approach incorporating both textitperiodic aggregation and (direct/error-feedback) compression mechanisms. Specifically, we consider compressed federated Q-learning with a generative model setup, where a central server learns an optimal Q-function by periodically aggregating compressed Q-estimates from local agents. For the first time, we characterize the impact of these two mechanisms (which have remained elusive) by providing a finite-time analysis of our algorithm, demonstrating strong convergence behaviors when utilizing either direct or error-feedback compression. Our bounds indicate improved solution accuracy concerning the number of agents and other federated hyperparameters while simultaneously reducing communication costs. To corroborate our theory, we also conduct in-depth numerical experiments to verify our findings, considering Top-K and Sparsified-K sparsification operators.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 43 of its 70 references that are in the map. First posted on arXiv 26 Mar 2024.

Builds on

43 of 70 references.

References · 43

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  2. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  3. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  4. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 561
  5. 2018Sparsified SGD with Memorycited by 373
  6. 2019Error Feedback Fixes SignSGD and other Gradient Compression Schemescited by 306
  7. 2017TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learningcited by 253
  8. 2018The Convergence of Sparsified Gradient Methodscited by 228
  9. 2018A Finite Time Analysis of Temporal Difference Learning With Linear Function Approximationcited by 225
  10. 2019Finite-Time Error Bounds For Linear Stochastic Approximation and TD Learningcited by 165
  11. 2020On Biased Compression for Distributed Learningcited by 140
  12. 2018ATOMO: Communication-efficient Learning via Atomic Sparsificationcited by 135
31 more
  1. 2021Linear Convergence in Federated Learning: Tackling Client Heterogeneity and Sparse Gradientscited by 122
  2. 2018Supervised Reinforcement Learning with Recurrent Neural Network for Dynamic Treatment Recommendationcited by 101
  3. 2020Finite-Time Analysis of Asynchronous Stochastic Approximation and Q-Learningcited by 93
  4. 2020Sample Complexity of Asynchronous Q-Learning: Sharper Analysis and Variance Reductioncited by 93
  5. 2021Is Q-Learning Minimax Optimal? A Tight Sample Complexity Analysiscited by 89
  6. 2019Stochastic approximation with cone-contractive operators: Sharp ℓ_∞-bounds for Q-learningcited by 74
  7. 2020Linearly Converging Error Compensated SGDcited by 61
  8. 2022Federated Reinforcement Learning with Environment Heterogeneitycited by 61
  9. 2021A Lyapunov Theory for Finite-Sample Guarantees of Asynchronous Q-Learning and TD-Learning Variantscited by 57
  10. 2021Fault-Tolerant Federated Reinforcement Learning with Theoretical Guaranteecited by 52
  11. 2023The Blessing of Heterogeneity in Federated Q-Learning: Linear Speedup and Beyondcited by 34
  12. 2024Finite-Time Analysis of On-Policy Heterogeneous Federated Reinforcement Learningcited by 29
  13. 2022Federated Stochastic Approximation under Markov Noise and Heterogeneity: Applications in Reinforcement Learningcited by 27
  14. 2021Sample and Communication-Efficient Decentralized Actor-Critic Algorithms with Finite-Time Analysiscited by 25
  15. 2023Federated Temporal Difference Learning with Linear Function Approximation under Environmental Heterogeneitycited by 25
  16. 2019RATQ: A Universal Fixed-Length Quantizer for Stochastic Optimizationcited by 23
  17. 2020Towards Understanding Asynchronous Advantage Actor-critic: Convergence and Linear Speedupcited by 23
  18. 2023Federated Q-Learning: Linear Regret Speedup with Low Communication Costcited by 19
  19. 2019Characterizing the Exact Behaviors of Temporal Difference Learning Algorithms Using Markov Jump Linear System Theorycited by 17
  20. 2019Gossip-based Actor-Learner Architectures for Deep Reinforcement Learningcited by 16
  21. 2019Finite-Time Performance of Distributed Temporal Difference Learning with Linear Function Approximationcited by 14
  22. 2023Federated TD Learning over Finite-Rate Erasure Channels: Linear Speedup under Markovian Samplingcited by 14
  23. 2018Convergent Actor-Critic Algorithms Under Off-Policy Training and Function Approximationcited by 12
  24. 2023Temporal Difference Learning with Compressed Updates: Error-Feedback meets Reinforcement Learningcited by 12
  25. 2020On the Convergence of Consensus Algorithms with Markovian Noise and Gradient Biascited by 10
  26. 2019Compressed Gradient Methods with Hessian-Aided Error Compensationcited by 6
  27. 2020Differentially Quantized Gradient Methodscited by 6
  28. 2021Multi-Agent Off-Policy TD Learning: Finite-Time Analysis with Near-Optimal Sample Complexity and Communication Complexitycited by 5
  29. 2024Parallel Momentum Methods Under Biased Gradient Estimationscited by 2
  30. 2024On the Convergence of Federated Learning Algorithms without Data Similaritycited by 2
  31. 2023Human-Inspired Framework to Accelerate Reinforcement Learningcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 1.

5 papers.

  1. 2025Gap-Dependent Bounds for Federated Q-learningcited by 4
  2. 2025Collaborative Value Function Estimation Under Model Mismatch: A Federated Temporal Difference Analysiscited by 2
  3. 2025Regret-Optimal Q-Learning with Low Cost for Single-Agent and Federated Reinforcement Learningcited by 2
  4. 2025Achieving Tighter Finite-Time Rates for Heterogeneous Federated Stochastic Approximation under Markovian Samplingcited by 1
  5. 2026Personalized Federated Reinforcement Learning via Model-Agnostic Meta-Learning: Convergence of Exact and Hessian-Free Meta-Policy Gradientscited by 0