On Global Convergence Rates for Federated Softmax Policy Gradient under Heterogeneous Environments
On the map: Deep reinforcement learning methods → RL for energy, networks and operations
Abstract (arXiv)
We provide global convergence rates for vanilla and entropy-regularized federated softmax stochastic policy gradient (FedPG) with local training. We show that FedPG converges to a near-optimal policy in terms of the average agent value, with a gap controlled by the level of heterogeneity. Remarkably, we obtain the first convergence rates for entropy-regularized policy gradient with explicit constants, leveraging a projection-like operator. Our results build upon a new analysis of federated averaging for non-convex objectives, based on the observation that the Lojasiewicz-type inequalities from the single-agent setting (Mei et al., 2020) do not hold for the federated objective. This uncovers a fundamental difference between single-agent and federated reinforcement learning: while single-agent optimal policies can be deterministic, federated objectives may inherently require stochastic policies.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in RL for energy, networks and operations, one of 15 directions in Deep reinforcement learning methods. First posted on arXiv 29 May 2025.
Builds on
0 of 0 references.
No references here yet.
Built on by
Citing papers per year: 2025: 1; 2026: 2.
3 papers.
- 2025Convergence Guarantees for Federated SARSA with Local Training and Heterogeneous Agentscited by 2
- 2026Refined Analysis of Entropy-Regularized Actor-Criticcited by 1
- 2026Personalized Federated Reinforcement Learning via Model-Agnostic Meta-Learning: Convergence of Exact and Hessian-Free Meta-Policy Gradientscited by 0
- 2026Personalized Federated Reinforcement Learning via Model-Agnostic Meta-Learning: Convergence of Exact and Hessian-Free Meta-Policy Gradients19 Sep 2026
- 2026Refined Analysis of Entropy-Regularized Actor-Critic23 May 2026
- 2025Convergence Guarantees for Federated SARSA with Local Training and Heterogeneous Agents19 Dec 2025
Similar, not cited
Close in wording, no citation either way
- 2024Towards Fast Rates for Federated and Multi-Task Reinforcement Learning
- 2024Finite-Time Analysis of On-Policy Heterogeneous Federated Reinforcement Learning
- 2025Achieving Tighter Finite-Time Rates for Heterogeneous Federated Stochastic Approximation under Markovian Sampling
- 2024Federated Reinforcement Learning with Constraint Heterogeneity
- 2025Federated Reinforcement Learning in Heterogeneous Environments
- 2024Global Convergence Guarantees for Federated Policy Gradient Methods with Adversaries
- 2024Momentum for the Win: Collaborative Federated Reinforcement Learning across Heterogeneous Environments
- 2024On the Convergence Rates of Federated Q-Learning across Heterogeneous Environments