mlsanityData to 14 Oct 2026

Federated Reinforcement Learning with Constraint Heterogeneity

arXiv 2405.03236 ↗First posted 6 May 2024cs.LG stat.MLcited by 2

On the map: Deep reinforcement learning methods → RL for energy, networks and operations

Abstract (arXiv)

We study a Federated Reinforcement Learning (FedRL) problem with constraint heterogeneity. In our setting, we aim to solve a reinforcement learning problem with multiple constraints while N training agents are located in N different environments with limited access to the constraint signals and they are expected to collaboratively learn a policy satisfying all constraint signals. Such learning problems are prevalent in scenarios of Large Language Model (LLM) fine-tuning and healthcare applications. To solve the problem, we propose federated primal-dual policy optimization methods based on traditional policy gradient methods. Specifically, we introduce N local Lagrange functions for agents to perform local policy updates, and these agents are then scheduled to periodically communicate on their local policies. Taking natural policy gradient (NPG) and proximal policy optimization (PPO) as policy optimization methods, we mainly focus on two instances of our algorithms, ie, FedNPG and FedPPO. We show that FedNPG achieves global convergence with an tildeO(1/√T) rate, and FedPPO efficiently solves complicated learning tasks with the use of deep neural networks.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in RL for energy, networks and operations, one of 15 directions in Deep reinforcement learning methods. Builds on 26 of its 37 references that are in the map. First posted on arXiv 6 May 2024.

Builds on

26 of 37 references.

References · 26

  1. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  2. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
  3. 2023Bias and Fairness in Large Language Models: A Surveycited by 639
  4. 2018Reward Constrained Policy Optimizationcited by 417
  5. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 337
  6. 2016Contextual Decision Processes with Low Bellman Rank are PAC-Learnablecited by 292
  7. 2019Neural Policy Gradient Methods: Global Optimality and Rates of Convergencecited by 159
  8. 2019Constrained Reinforcement Learning Has Zero Duality Gapcited by 151
  9. 2020CRPO: A New Approach for Safe Reinforcement Learning with Convergence Guaranteecited by 134
  10. 2013Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)cited by 111
  11. 2017Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learningcited by 98
  12. 2019Model-free Deep Reinforcement Learning for Urban Autonomous Drivingcited by 98
14 more
  1. 2019Lifelong Federated Reinforcement Learning: A Learning Architecture for Navigation in Cloud Robotic Systemscited by 78
  2. 2022Constrained Variational Policy Optimization for Safe Reinforcement Learningcited by 71
  3. 2022Federated Reinforcement Learning with Environment Heterogeneitycited by 63
  4. 2018Accelerated Primal-Dual Policy Optimization for Safe Reinforcement Learningcited by 62
  5. 2019Value constrained model-free continuous controlcited by 34
  6. 2015Safety-Constrained Reinforcement Learning for MDPscited by 33
  7. 2023Integration of Large Language Models and Federated Learningcited by 31
  8. 2021Density Constrained Reinforcement Learningcited by 25
  9. 2022Scaling Language Model Size in Cross-Device Federated Learningcited by 25
  10. 2021Policy Optimization for Constrained MDPs with Provable Fast Global Convergencecited by 23
  11. 2021Faster Algorithm and Sharper Analysis for Constrained Markov Decision Processcited by 19
  12. 2021Finite-Time Complexity of Online Primal-Dual Natural Actor-Critic Algorithm for Constrained Markov Decision Processescited by 17
  13. 2018Fingerprint Policy Optimisation for Robust Reinforcement Learningcited by 7
  14. 2016Robust Hybrid Learning for Estimating Personalized Dynamic Treatment Regimenscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Decoupled Training with Local Reinforcement Fine-Tuning in Federated Learningcited by 0
  2. 2026Constraint-Aware Aggregation for Federated Reinforcement Learning in Microgrid Energy Coordinationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Finite-Time Analysis of On-Policy Heterogeneous Federated Reinforcement Learning
  2. 2025Federated Reinforcement Learning in Heterogeneous Environments
  3. 2024Momentum for the Win: Collaborative Federated Reinforcement Learning across Heterogeneous Environments
  4. 2025FedHPD: Heterogeneous Federated Reinforcement Learning via Policy Distillation
  5. 2025On Global Convergence Rates for Federated Softmax Policy Gradient under Heterogeneous Environments
  6. 2024Towards Fast Rates for Federated and Multi-Task Reinforcement Learning
  7. 2024Asynchronous Federated Reinforcement Learning with Policy Gradient Updates: Algorithm Design and Convergence Analysis
  8. 2026FedGuide: Diffusion Prior Alignment and Value Baseline Guidance for Heterogeneous Federated Reinforcement Learning