mlsanityData to 14 Oct 2026

Learning from Unreliable Trajectories: Adversarially-Robust Federated Q-Learning

arXiv 2610.06918 ↗First posted 2 Oct 2026cs.LG cs.SY eess.SYcited by 0

On the map: Deep reinforcement learning methods → Adversarial robustness and RL in finance also fits: RL theory and sample complexity

Abstract (arXiv)

We study federated reinforcement learning in which multiple agents interact with a common Markov decision process and communicate through a central server to collaboratively learn the optimal state-action value function. Our goal is to understand whether the sample-efficiency benefits of collaboration can be retained when a fraction of the agents behave adversarially and transmit arbitrarily corrupted information. To address this problem, we introduce Robust Async-Fed-Q, an epoch-based federated learning algorithm that combines variance-reduced estimation of the Bellman optimality operator at the agents with robust aggregation at the server. We establish high-probability finite-time guarantees showing that the proposed method preserves the statistical gains of collaboration among the honest agents while tolerating adversarial corruption. In particular, the effect of the adversarial agents decreases as the amount of data collected by each honest agent grows and eventually vanishes in the infinite-sample limit. We complement these guarantees with information-theoretic lower bounds that characterize the unavoidable statistical cost of adversarial corruption, leading to the first nearly matching upper and lower bounds for adversarially robust federated reinforcement learning. We further extend our framework to accommodate single-trajectory Markovian sampling and heterogeneous partial coverage, where different agents may explore different regions of the state-action space and learning relies on their collective coverage. Finally, our epoch-based design substantially improves the best known communication complexity for federated Q-learning under asynchronous sampling.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Adversarial robustness and RL in finance, one of 15 directions in Deep reinforcement learning methods. Builds on 34 of its 56 references that are in the map. First posted on arXiv 2 Oct 2026.

Builds on

34 of 56 references.

References · 34

  1. 2018Byzantine-Robust Distributed Learning: Towards Optimal Statistical Ratescited by 711
  2. 2016Federated Optimization: Distributed Machine Learning for On-Device Intelligencecited by 564
  3. 2019Robust Aggregation for Federated Learningcited by 317
  4. 2018A Finite Time Analysis of Temporal Difference Learning With Linear Function Approximationcited by 229
  5. 2020Finite-Time Analysis of Asynchronous Stochastic Approximation and Q-Learningcited by 96
  6. 2021Is Q-Learning Minimax Optimal? A Tight Sample Complexity Analysiscited by 92
  7. 2021Federated Reinforcement Learning: Techniques, Applications, and Open Challengescited by 81
  8. 2019Stochastic approximation with cone-contractive operators: Sharp ℓ_∞-bounds for Q-learningcited by 77
  9. 2018DRACO: Byzantine-resilient Distributed Training via Redundant Gradientscited by 75
  10. 2022Federated Reinforcement Learning with Environment Heterogeneitycited by 63
  11. 2021Fault-Tolerant Federated Reinforcement Learning with Theoretical Guaranteecited by 54
  12. 2022Adapting to Mixing Time in Stochastic Optimization with Markovian Datacited by 36
22 more
  1. 2023The Blessing of Heterogeneity in Federated Q-Learning: Linear Speedup and Beyondcited by 36
  2. 2024Finite-Time Analysis of On-Policy Heterogeneous Federated Reinforcement Learningcited by 31
  3. 2023Federated Temporal Difference Learning with Linear Function Approximation under Environmental Heterogeneitycited by 27
  4. 2020Towards Understanding Asynchronous Advantage Actor-critic: Convergence and Linear Speedupcited by 25
  5. 2014On TD(0) with function approximation: Concentration bounds and a centered variant with exponential convergencecited by 22
  6. 2023Improved Communication Efficiency in Federated Natural Policy Gradient via ADMM-based Gradient Updatescited by 21
  7. 2016Distributed Optimization Under Adversarial Nodescited by 20
  8. 2023Federated TD Learning over Finite-Rate Erasure Channels: Linear Speedup under Markovian Samplingcited by 15
  9. 2023Distributed TD(0) with Almost No Communicationcited by 13
  10. 2022Private and Byzantine-Proof Cooperative Decision-Makingcited by 10
  11. 2024Federated Q-Learning with Reference-Advantage Decomposition: Almost Optimal Regret and Logarithmic Communication Costcited by 10
  12. 2024The Sample-Communication Complexity Trade-off in Federated Q-Learningcited by 9
  13. 2017Linear Stochastic Approximation: Constant Step-Size and Iterate Averagingcited by 8
  14. 2022Byzantine-Robust Federated Linear Banditscited by 8
  15. 2024Momentum for the Win: Collaborative Federated Reinforcement Learning across Heterogeneous Environmentscited by 8
  16. 2024SCAFFLSA: Taming Heterogeneity in Federated Linear Stochastic Approximation and TD Learningcited by 7
  17. 2024Compressed Federated Reinforcement Learning with a Generative Modelcited by 6
  18. 2024Federated UCBVI: Communication-Efficient Federated Regret Minimization with Heterogeneous Agentscited by 6
  19. 2022Collaborative Linear Bandits with Adversarial Agents: Near-Optimal Regret Boundscited by 5
  20. 2024One-Shot Averaging for Distributed TD(λ) Under Markov Samplingcited by 5
  21. 2024Towards Fast Rates for Federated and Multi-Task Reinforcement Learningcited by 4
  22. 2026Robust Federated Q-Learning with Almost No Communicationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Global Convergence Guarantees for Federated Policy Gradient Methods with Adversaries
  2. 2026Provably Efficient Federated Reinforcement Learning with Linear Function Approximation and Logarithmic Communication Cost
  3. 2025Provably Robust Federated Reinforcement Learning
  4. 2026FedQHD: Closed-Form Function-Space Federated Reinforcement Learning
  5. 2026Robust Asynchronous Q-Learning under Reward and State Corruption via Batching
  6. 2024Federated Offline Reinforcement Learning: Collaborative Single-Policy Coverage Suffices
  7. 2025Collaborative Value Function Estimation Under Model Mismatch: A Federated Temporal Difference Analysis
  8. 2025Corruption-Tolerant Asynchronous Q-Learning with Near-Optimal Rates