mlsanityData to 7 Oct 2026

On the Linear Speedup of Personalized Federated Reinforcement Learning with Shared Representations

arXiv 2411.15014 ↗First posted 22 Nov 2024cs.LG math.OC stat.MLcited by 4

Not on the map. Why?

Abstract (arXiv)

Federated reinforcement learning (FedRL) enables multiple agents to collaboratively learn a policy without sharing their local trajectories collected during agent-environment interactions. However, in practice, the environments faced by different agents are often heterogeneous, leading to poor performance by the single policy learned by existing FedRL algorithms on individual agents. In this paper, we take a further step and introduce a personalized FedRL framework (PFedRL) by taking advantage of possibly shared common structure among agents in heterogeneous environments. Specifically, we develop a class of PFedRL algorithms named PFedRL-Rep that learns (1) a shared feature representation collaboratively among all agents, and (2) an agent-specific weight vector personalized to its local environment. We analyze the convergence of PFedTD-Rep, a particular instance of the framework with temporal difference (TD) learning and linear representations. To the best of our knowledge, we are the first to prove a linear convergence speedup with respect to the number of agents in the PFedRL setting. To achieve this, we show that PFedTD-Rep is an example of the federated two-timescale stochastic approximation with Markovian noise. Experimental results demonstrate that PFedTD-Rep, along with an extension to the control setting based on deep Q-networks (DQN), not only improve learning in heterogeneous settings, but also provide better generalization to new environments.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 43 of its 61 references that are in the map. First posted on arXiv 22 Nov 2024.

Builds on

43 of 61 references.

References · 43

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  3. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  4. 2017Federated Multi-Task Learningcited by 669
  5. 2018Federated Learning for Mobile Keyboard Predictioncited by 565
  6. 2019Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithmscited by 555
  7. 2021Exploiting Shared Representations for Personalized Federated Learningcited by 537
  8. 2019Federated Learning with Personalization Layerscited by 533
  9. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 514
  10. 2020Personalized Federated Learning: A Meta-Learning Approachcited by 309
  11. 2021Federated Learning for Internet of Things: A Comprehensive Surveycited by 247
  12. 2020FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPscited by 232
31 more
  1. 2018A Finite Time Analysis of Temporal Difference Learning With Linear Function Approximationcited by 225
  2. 2018Applied Federated Learning: Improving Google Keyboard Query Suggestionscited by 202
  3. 2019Finite-Time Error Bounds For Linear Stochastic Approximation and TD Learningcited by 165
  4. 2018Federated Meta-Learning with Fast Convergence and Efficient Communicationcited by 154
  5. 2021Model-free Representation Learning and Exploration in Low-rank MDPscited by 82
  6. 2021Federated Reinforcement Learning: Techniques, Applications, and Open Challengescited by 79
  7. 2019Lifelong Federated Reinforcement Learning: A Learning Architecture for Navigation in Cloud Robotic Systemscited by 77
  8. 2022Federated Reinforcement Learning with Environment Heterogeneitycited by 61
  9. 2021Fault-Tolerant Federated Reinforcement Learning with Theoretical Guaranteecited by 52
  10. 2022Efficient Reinforcement Learning in Block MDPs: A Model-free Representation Learning Approachcited by 51
  11. 2021Federated Learning for Internet of Things: Applications, Challenges, and Opportunitiescited by 49
  12. 2020Nonlinear Two-Time-Scale Stochastic Approximation: Convergence and Finite-Time Performancecited by 35
  13. 2023The Blessing of Heterogeneity in Federated Q-Learning: Linear Speedup and Beyondcited by 34
  14. 2024Finite-Time Analysis of On-Policy Heterogeneous Federated Reinforcement Learningcited by 29
  15. 2022A Multi-agent Reinforcement Learning Approach for Efficient Client Selection in Federated Learningcited by 28
  16. 2023Federated Temporal Difference Learning with Linear Function Approximation under Environmental Heterogeneitycited by 25
  17. 2020Towards Understanding Asynchronous Advantage Actor-critic: Convergence and Linear Speedupcited by 23
  18. 2024Asynchronous Federated Reinforcement Learning with Policy Gradient Updates: Algorithm Design and Convergence Analysiscited by 21
  19. 2020Federated Learning for Resource-Constrained IoT Devices: Panoramas and State-of-the-artcited by 19
  20. 2023Federated Q-Learning: Linear Regret Speedup with Low Communication Costcited by 19
  21. 2022Towards Fair Federated Recommendation Learning: Characterizing the Inter-Dependence of System and Data Heterogeneitycited by 17
  22. 2023Federated TD Learning over Finite-Rate Erasure Channels: Linear Speedup under Markovian Samplingcited by 14
  23. 2024Federated Offline Reinforcement Learning: Collaborative Single-Policy Coverage Sufficescited by 11
  24. 2021Finite-Time Convergence Rates of Nonlinear Two-Time-Scale Stochastic Approximation under Markovian Noisecited by 10
  25. 2022Straggler-Resilient Personalized Federated Learningcited by 9
  26. 2024Fast Two-Time-Scale Stochastic Gradient Method with Applications in Reinforcement Learningcited by 8
  27. 2024The Sample-Communication Complexity Trade-off in Federated Q-Learningcited by 8
  28. 2022Personalized Federated Learning with Communication Compressioncited by 6
  29. 2023Private Learning with Public Featurescited by 6
  30. 2023DePRL: Achieving Linear Convergence Speedup in Personalized Decentralized Learning with Shared Representationscited by 5
  31. 2023Now It Sounds Like You: Learning Personalized Vocabulary On Devicecited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2026FedMOA: Federated GRPO for Personalized Reasoning LLMs under Heterogeneous Rewardscited by 2
  2. 2025Personalized Collaborative Learning with Affinity-Based Variance Reductioncited by 1
  3. 2026Decoupled Training with Local Reinforcement Fine-Tuning in Federated Learningcited by 0
  4. 2026Personalized Federated Reinforcement Learning via Model-Agnostic Meta-Learning: Convergence of Exact and Hessian-Free Meta-Policy Gradientscited by 0