mlsanityData to 14 Oct 2026

Single-Loop Federated Actor-Critic across Heterogeneous Environments

arXiv 2412.14555 ↗First posted 19 Dec 2024cs.LG cs.DC cs.MAcited by 0

On the map: Deep reinforcement learning methods → RL for energy, networks and operations

Abstract (arXiv)

Federated reinforcement learning (FRL) has emerged as a promising paradigm, enabling multiple agents to collaborate and learn a shared policy adaptable across heterogeneous environments. Among the various reinforcement learning (RL) algorithms, the actor-critic (AC) algorithm stands out for its low variance and high sample efficiency. However, little to nothing is known theoretically about AC in a federated manner, especially each agent interacts with a potentially different environment. The lack of such results is attributed to various technical challenges: a two-level structure illustrating the coupling effect between the actor and the critic, heterogeneous environments, Markovian sampling and multiple local updates. In response, we study textitSingle-loop Federated Actor Critic (SFAC) where agents perform actor-critic learning in a two-level federated manner while interacting with heterogeneous environments. We then provide bounds on the convergence error of SFAC. The results show that the convergence error asymptotically converges to a near-stationary point, with the extent proportional to environment heterogeneity. Moreover, the sample complexity exhibits a linear speed-up through the federation of agents. We evaluate the performance of SFAC through numerical experiments using common RL benchmarks, which demonstrate its effectiveness.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL for energy, networks and operations, one of 15 directions in Deep reinforcement learning methods. Builds on 19 of its 34 references that are in the map. First posted on arXiv 19 Dec 2024.

Builds on

19 of 34 references.

References · 19

  1. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 906
  2. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
  3. 2016Safe, Multi-Agent, Reinforcement Learning for Autonomous Drivingcited by 311
  4. 2018Approximation Methods for Bilevel Programmingcited by 236
  5. 2018Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agentscited by 194
  6. 2019Finite-Time Error Bounds For Linear Stochastic Approximation and TD Learningcited by 166
  7. 2019Neural Policy Gradient Methods: Global Optimality and Rates of Convergencecited by 159
  8. 2020A Two-Timescale Framework for Bilevel Optimization: Complexity Analysis and Application to Actor-Criticcited by 144
  9. 2022Federated Reinforcement Learning with Environment Heterogeneitycited by 63
  10. 2022FedNest: Federated Bilevel, Minimax, and Compositional Optimizationcited by 60
  11. 2021Fault-Tolerant Federated Reinforcement Learning with Theoretical Guaranteecited by 54
  12. 2022Finite-time analysis of single-timescale actor-criticcited by 32
7 more
  1. 2020When Deep Reinforcement Learning Meets Federated Learning: Intelligent Multi-Timescale Resource Management for Multi-access Edge Computing in 5G Ultra Dense Networkcited by 30
  2. 2022FedKL: Tackling Data Heterogeneity in Federated Reinforcement Learning by Penalizing KL Divergencecited by 29
  3. 2023Federated Temporal Difference Learning with Linear Function Approximation under Environmental Heterogeneitycited by 27
  4. 2020A Decentralized Policy Gradient Approach to Multi-task Reinforcement Learningcited by 25
  5. 2020Towards Understanding Asynchronous Advantage Actor-critic: Convergence and Linear Speedupcited by 25
  6. 2019Decentralized Multi-Agent Reinforcement Learning with Networked Agents: Recent Advancescited by 23
  7. 2023Achieving Linear Speedup in Non-IID Federated Bilevel Learningcited by 18

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Federated Reinforcement Learning in Heterogeneous Environments
  2. 2024Finite-Time Analysis of On-Policy Heterogeneous Federated Reinforcement Learning
  3. 2024Momentum for the Win: Collaborative Federated Reinforcement Learning across Heterogeneous Environments
  4. 2026Collaborative Yet Personalized Policy Training: Single-Timescale Federated Actor-Critic
  5. 2025The Actor-Critic Update Order Matters for PPO in Federated Reinforcement Learning
  6. 2024CAESAR: Enhancing Federated RL in Heterogeneous MDPs through Convergence-Aware Sampling with Screening
  7. 2024On the Convergence Rates of Federated Q-Learning across Heterogeneous Environments
  8. 2025FedHPD: Heterogeneous Federated Reinforcement Learning via Policy Distillation