mlsanityData to 14 Oct 2026

Federated Q-Learning with Reference-Advantage Decomposition: Almost Optimal Regret and Logarithmic Communication Cost

arXiv 2405.18795 ↗First posted 29 May 2024stat.ML cs.LGcited by 10

On the map: Deep reinforcement learning methods → RL for energy, networks and operations

Abstract (arXiv)

In this paper, we consider model-free federated reinforcement learning for tabular episodic Markov decision processes. Under the coordination of a central server, multiple agents collaboratively explore the environment and learn an optimal policy without sharing their raw data. Despite recent advances in federated Q-learning algorithms achieving near-linear regret speedup with low communication cost, existing algorithms only attain suboptimal regrets compared to the information bound. We propose a novel model-free federated Q-learning algorithm, termed FedQ-Advantage. Our algorithm leverages reference-advantage decomposition for variance reduction and operates under two distinct mechanisms: synchronization between the agents and the server, and policy update, both triggered by events. We prove that our algorithm not only requires a lower logarithmic communication cost but also achieves an almost optimal regret, reaching the information bound up to a logarithmic factor and near-linear regret speedup compared to its single-agent counterpart when the time horizon is sufficiently large.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in RL for energy, networks and operations, one of 15 directions in Deep reinforcement learning methods. First posted on arXiv 29 May 2024.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 4; 2026: 2.

10 papers.

  1. 2024Gap-Dependent Bounds for Q-Learning using Reference-Advantage Decompositioncited by 8
  2. 2024In-Trajectory Inverse Reinforcement Learning: Learn Incrementally Before An Ongoing Trajectory Terminatescited by 7
  3. 2024Federated UCBVI: Communication-Efficient Federated Regret Minimization with Heterogeneous Agentscited by 6
  4. 2025Gap-Dependent Bounds for Federated Q-learningcited by 4
  5. 2025Regret-Optimal Q-Learning with Low Cost for Single-Agent and Federated Reinforcement Learningcited by 2
  6. 2024Understanding the Statistical Accuracy-Communication Trade-off in Personalized Federated Learning with Minimax Guaranteescited by 1
  7. 2025Q-Learning with Fine-Grained Gap-Dependent Regretcited by 1
  8. 2025Explainable reinforcement learning from human feedback to improve alignmentcited by 1
  9. 2026Provably Efficient Federated Reinforcement Learning with Linear Function Approximation and Logarithmic Communication Costcited by 1
  10. 2026Learning from Unreliable Trajectories: Adversarially-Robust Federated Q-Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024The Sample-Communication Complexity Trade-off in Federated Q-Learning
  2. 2024Federated Offline Reinforcement Learning: Collaborative Single-Policy Coverage Suffices
  3. 2026Robust Federated Q-Learning with Almost No Communication
  4. 2024Compressed Federated Reinforcement Learning with a Generative Model
  5. 2023Federated TD Learning over Finite-Rate Erasure Channels: Linear Speedup under Markovian Sampling
  6. 2024On the Convergence Rates of Federated Q-Learning across Heterogeneous Environments
  7. 2024Towards Fast Rates for Federated and Multi-Task Reinforcement Learning
  8. 2025Parameter-Free Federated TD Learning with Markov Noise in Heterogeneous Environments