mlsanityData to 14 Oct 2026

Robust Federated Q-Learning with Almost No Communication

arXiv 2609.20174 ↗First posted 24 Aug 2026cs.LG cs.SY eess.SYcited by 1

On the map: Deep reinforcement learning methods → RL for energy, networks and operations

Abstract (arXiv)

We consider a federated reinforcement learning setting involving M agents, all of whom interact with a common Markov Decision Process (MDP). The agents exchange information via a central server to learn the optimal value function. Our goal is to understand to what extent one can hope for collaborative sample-complexity speedups in such a setting, when a small fraction of the agents are adversarial and can act arbitrarily. To that end, we propose Robust Fed-Q, a federated Q-learning algorithm that blends ideas from both model-based and model-free RL, along with the median-of-means device from robust statistics. We prove that despite corruption, with high-probability, Robust Fed-Q (i) guarantees exact convergence to the optimal value function in the limit of infinite samples, and (ii) enjoys near-optimal finite-time rates that benefit from collaboration. In addition, our approach requires just tildeO(1) rounds of communication to achieve each of the above guarantees, a feature of independent interest in FL where communication is the major bottleneck.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in RL for energy, networks and operations, one of 15 directions in Deep reinforcement learning methods. Builds on 13 of its 27 references that are in the map. First posted on arXiv 24 Aug 2026.

Builds on

13 of 27 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Global Convergence Guarantees for Federated Policy Gradient Methods with Adversaries15 Mar

Other references · 12

  1. 2018Byzantine-Robust Distributed Learning: Towards Optimal Statistical Ratescited by 711
  2. 2016Federated Optimization: Distributed Machine Learning for On-Device Intelligencecited by 564
  3. 2019Mean estimation and regression under heavy-tailed distributions--a surveycited by 136
  4. 2020Finite-Time Analysis of Asynchronous Stochastic Approximation and Q-Learningcited by 96
  5. 2021Is Q-Learning Minimax Optimal? A Tight Sample Complexity Analysiscited by 92
  6. 2021Federated Reinforcement Learning: Techniques, Applications, and Open Challengescited by 81
  7. 2019Stochastic approximation with cone-contractive operators: Sharp ℓ_∞-bounds for Q-learningcited by 77
  8. 2018DRACO: Byzantine-resilient Distributed Training via Redundant Gradientscited by 75
  9. 2022Federated Reinforcement Learning with Environment Heterogeneitycited by 63
  10. 2023The Blessing of Heterogeneity in Federated Q-Learning: Linear Speedup and Beyondcited by 36
  11. 2023Federated Temporal Difference Learning with Linear Function Approximation under Environmental Heterogeneitycited by 27
  12. 2022Collaborative Linear Bandits with Adversarial Agents: Near-Optimal Regret Boundscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Learning from Unreliable Trajectories: Adversarially-Robust Federated Q-Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024The Sample-Communication Complexity Trade-off in Federated Q-Learning
  2. 2024Federated Q-Learning with Reference-Advantage Decomposition: Almost Optimal Regret and Logarithmic Communication Cost
  3. 2024Compressed Federated Reinforcement Learning with a Generative Model
  4. 2026Provably Efficient Federated Reinforcement Learning with Linear Function Approximation and Logarithmic Communication Cost
  5. 2026FedQHD: Closed-Form Function-Space Federated Reinforcement Learning
  6. 2024Federated Offline Reinforcement Learning: Collaborative Single-Policy Coverage Suffices
  7. 2024On the Convergence Rates of Federated Q-Learning across Heterogeneous Environments
  8. 2025Provably Robust Federated Reinforcement Learning