mlsanityData to 14 Oct 2026

Client Selection for Federated Policy Optimization with Environment Heterogeneity

arXiv 2305.10978 ↗First posted 18 May 2023cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

The development of Policy Iteration (PI) has inspired many recent algorithms for Reinforcement Learning (RL), including several policy gradient methods that gained both theoretical soundness and empirical success on a variety of tasks. The theory of PI is rich in the context of centralized learning, but its study under the federated setting is still in the infant stage. This paper investigates the federated version of Approximate PI (API) and derives its error bound, taking into account the approximation error introduced by environment heterogeneity. We theoretically prove that a proper client selection scheme can reduce this error bound. Based on the theoretical result, we propose a client selection algorithm to alleviate the additional approximation error caused by environment heterogeneity. Experiment results show that the proposed algorithm outperforms other biased and unbiased client selection methods on the federated mountain car problem, the Mujoco Hopper problem, and the SUMO-based autonomous vehicle training problem by effectively selecting clients with a lower level of heterogeneity from the population distribution.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 37 of its 71 references that are in the map. First posted on arXiv 18 May 2023.

Builds on

37 of 71 references.

References · 37

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  3. 2019Advances and Open Problems in Federated Learningcited by 3,046
  4. 2016Deep Learning with Differential Privacycited by 2,728
  5. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  6. 2015Trust Region Policy Optimizationcited by 2,580
  7. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  8. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  9. 2016OpenAI Gymcited by 1,736
  10. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,229
  11. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  12. 2019Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithmscited by 567
25 more
  1. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
  2. 2019Federated Learning with Differential Privacy: Algorithms and Performance Analysiscited by 536
  3. 2019Fair Resource Allocation in Federated Learningcited by 394
  4. 2018Client Selection for Federated Learning with Heterogeneous Resources in Mobile Edgecited by 352
  5. 2019Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networkscited by 222
  6. 2018Top-K Off-Policy Correction for a REINFORCE Recommender Systemcited by 182
  7. 2019Neural Policy Gradient Methods: Global Optimality and Rates of Convergencecited by 159
  8. 2018cpSGD: Communication-efficient and differentially-private distributed SGDcited by 158
  9. 2017Revisiting Distributed Synchronous SGDcited by 142
  10. 2020Optimal Client Sampling for Federated Learningcited by 124
  11. 2018Stochastic Variance-Reduced Policy Gradientcited by 106
  12. 2015Massively Parallel Methods for Deep Reinforcement Learningcited by 82
  13. 2020Convergence Time Optimization for Federated Learning over Wireless Networkscited by 81
  14. 2021Federated Reinforcement Learning: Techniques, Applications, and Open Challengescited by 81
  15. 2022Federated Reinforcement Learning with Environment Heterogeneitycited by 63
  16. 2021Fault-Tolerant Federated Reinforcement Learning with Theoretical Guaranteecited by 54
  17. 2019Federated Transfer Reinforcement Learning for Autonomous Drivingcited by 44
  18. 2022FedKL: Tackling Data Heterogeneity in Federated Reinforcement Learning by Penalizing KL Divergencecited by 29
  19. 2018Expected Policy Gradients for Reinforcement Learningcited by 28
  20. 2020Steady State Analysis of Episodic Reinforcement Learningcited by 21
  21. 2019Deep Conservative Policy Iterationcited by 15
  22. 2022Federated Offline Reinforcement Learningcited by 13
  23. 2019Learning and Planning for Time-Varying MDPs Using Maximum Likelihood Estimationcited by 10
  24. 2021Client Selection in Federated Learning based on Gradients Importancecited by 9
  25. 2022Federated Reinforcement Learning for Collective Navigation of Robotic Swarmscited by 7

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 2.

3 papers.

  1. 2024Momentum for the Win: Collaborative Federated Reinforcement Learning across Heterogeneous Environmentscited by 8
  2. 2025FOVA: Offline Federated Reinforcement Learning with Mixed-Quality Datacited by 5
  3. 2025The Actor-Critic Update Order Matters for PPO in Federated Reinforcement Learningcited by 2