mlsanityData to 14 Oct 2026

Towards Fast Rates for Federated and Multi-Task Reinforcement Learning

arXiv 2409.05291 ↗First posted 9 Sep 2024cs.LG cs.SY eess.SYcited by 4

On the map: Deep reinforcement learning methods → RL for energy, networks and operations

Abstract (arXiv)

We consider a setting involving N agents, where each agent interacts with an environment modeled as a Markov Decision Process (MDP). The agents' MDPs differ in their reward functions, capturing heterogeneous objectives/tasks. The collective goal of the agents is to communicate intermittently via a central server to find a policy that maximizes the average of long-term cumulative rewards across environments. The limited existing work on this topic either only provide asymptotic rates, or generate biased policies, or fail to establish any benefits of collaboration. In response, we propose Fast-FedPG - a novel federated policy gradient algorithm with a carefully designed bias-correction mechanism. Under a gradient-domination condition, we prove that our algorithm guarantees (i) fast linear convergence with exact gradients, and (ii) sub-linear rates that enjoy a linear speedup w.r.t. the number of agents with noisy, truncated policy gradients. Notably, in each case, the convergence is to a globally optimal policy with no heterogeneity-induced bias. In the absence of gradient-domination, we establish convergence to a first-order stationary point at a rate that continues to benefit from collaboration.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in RL for energy, networks and operations, one of 15 directions in Deep reinforcement learning methods. Builds on 20 of its 26 references that are in the map. First posted on arXiv 9 Sep 2024.

Builds on

20 of 26 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Momentum for the Win: Collaborative Federated Reinforcement Learning across Heterogeneous Environments29 May
  2. 2024Finite-Time Analysis of On-Policy Heterogeneous Federated Reinforcement Learning27 Jan

Other references · 18

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  2. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 264
  3. 2018Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulatorcited by 250
  4. 2021Multi-Task Reinforcement Learning with Context-based Representationscited by 174
  5. 2021Linear Convergence in Federated Learning: Tackling Client Heterogeneity and Sparse Gradientscited by 123
  6. 2022On the Convergence Rates of Policy Gradient Methodscited by 119
  7. 2023Stochastic Controlled Averaging for Federated Learning with Communication Compressioncited by 101
  8. 2021Federated Reinforcement Learning: Techniques, Applications, and Open Challengescited by 81
  9. 2021A general sample complexity analysis of vanilla policy gradientcited by 76
  10. 2022Federated Reinforcement Learning with Environment Heterogeneitycited by 63
  11. 2023The Blessing of Heterogeneity in Federated Q-Learning: Linear Speedup and Beyondcited by 36
  12. 2022FedKL: Tackling Data Heterogeneity in Federated Reinforcement Learning by Penalizing KL Divergencecited by 29
6 more
  1. 2023Federated Temporal Difference Learning with Linear Function Approximation under Environmental Heterogeneitycited by 27
  2. 2020A Decentralized Policy Gradient Approach to Multi-task Reinforcement Learningcited by 25
  3. 2023Improved Communication Efficiency in Federated Natural Policy Gradient via ADMM-based Gradient Updatescited by 21
  4. 2023Federated TD Learning over Finite-Rate Erasure Channels: Linear Speedup under Markovian Samplingcited by 15
  5. 2023Distributed TD(0) with Almost No Communicationcited by 13
  6. 2024One-Shot Averaging for Distributed TD(λ) Under Markov Samplingcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025Achieving Tighter Finite-Time Rates for Heterogeneous Federated Stochastic Approximation under Markovian Samplingcited by 2
  2. 2025Convergence Guarantees for Federated SARSA with Local Training and Heterogeneous Agentscited by 2
  3. 2026Personalized Federated Reinforcement Learning via Model-Agnostic Meta-Learning: Convergence of Exact and Hessian-Free Meta-Policy Gradientscited by 0
  4. 2026Learning from Unreliable Trajectories: Adversarially-Robust Federated Q-Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025On Global Convergence Rates for Federated Softmax Policy Gradient under Heterogeneous Environments
  2. 2024Asynchronous Federated Reinforcement Learning with Policy Gradient Updates: Algorithm Design and Convergence Analysis
  3. 2024CAESAR: Enhancing Federated RL in Heterogeneous MDPs through Convergence-Aware Sampling with Screening
  4. 2024Federated Reinforcement Learning with Constraint Heterogeneity
  5. 2025Parameter-Free Federated TD Learning with Markov Noise in Heterogeneous Environments
  6. 2024On the Convergence Rates of Federated Q-Learning across Heterogeneous Environments
  7. 2024Global Convergence Guarantees for Federated Policy Gradient Methods with Adversaries
  8. 2024Momentum-Based Federated Reinforcement Learning with Interaction and Communication Efficiency