mlsanityData to 14 Oct 2026

FedHQL: Federated Heterogeneous Q-Learning

arXiv 2301.11135 ↗First posted 26 Jan 2023cs.LG cs.DCcited by 6

Not on the map. Why?

Abstract (arXiv)

Federated Reinforcement Learning (FedRL) encourages distributed agents to learn collectively from each other's experience to improve their performance without exchanging their raw trajectories. The existing work on FedRL assumes that all participating agents are homogeneous, which requires all agents to share the same policy parameterization (e.g., network architectures and training configurations). However, in real-world applications, agents are often in disagreement about the architecture and the parameters, possibly also because of disparate computational budgets. Because homogeneity is not given in practice, we introduce the problem setting of Federated Reinforcement Learning with Heterogeneous And bLack-box agEnts (FedRL-HALE). We present the unique challenges this new setting poses and propose the Federated Heterogeneous Q-Learning (FedHQL) algorithm that principally addresses these challenges. We empirically demonstrate the efficacy of FedHQL in boosting the sample efficiency of heterogeneous agents with distinct policy parameterization using standard RL tasks.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 22 of its 36 references that are in the map. First posted on arXiv 26 Jan 2023.

Builds on

22 of 36 references.

References · 22

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  3. 2019Advances and Open Problems in Federated Learningcited by 3,046
  4. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  5. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  6. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  7. 2020The Future of Digital Health with Federated Learningcited by 633
  8. 2016Federated Optimization: Distributed Machine Learning for On-Device Intelligencecited by 564
  9. 2018Distributed Prioritized Experience Replaycited by 269
  10. 2018Stochastic Variance-Reduced Policy Gradientcited by 106
  11. 2015Massively Parallel Methods for Deep Reinforcement Learningcited by 82
  12. 2019Lifelong Federated Reinforcement Learning: A Learning Architecture for Navigation in Cloud Robotic Systemscited by 78
10 more
  1. 2017Boltzmann Exploration Done Rightcited by 69
  2. 2022Federated Reinforcement Learning with Environment Heterogeneitycited by 63
  3. 2021Fault-Tolerant Federated Reinforcement Learning with Theoretical Guaranteecited by 54
  4. 2019Federated Transfer Reinforcement Learning for Autonomous Drivingcited by 44
  5. 2019Federated Deep Reinforcement Learningcited by 35
  6. 2020When Deep Reinforcement Learning Meets Federated Learning: Intelligent Multi-Timescale Resource Management for Multi-access Edge Computing in 5G Ultra Dense Networkcited by 30
  7. 2022Federated Stochastic Approximation under Markov Noise and Heterogeneity: Applications in Reinforcement Learningcited by 27
  8. 2016Collective Robot Reinforcement Learning with Distributed Asynchronous Guided Policy Searchcited by 25
  9. 2022Federated Neural Banditscited by 17
  10. 2022Byzantine-Robust Online and Offline Distributed Reinforcement Learningcited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 0; 2025: 4; 2026: 1.

6 papers.

  1. 2023Action and Trajectory Planning for Urban Autonomous Driving with Hierarchical Reinforcement Learningcited by 7
  2. 2025Gap-Dependent Bounds for Federated Q-learningcited by 4
  3. 2025Regret-Optimal Q-Learning with Low Cost for Single-Agent and Federated Reinforcement Learningcited by 2
  4. 2025Position Paper: Rethinking Privacy in RL for Sequential Decision-making in the Age of LLMscited by 0
  5. 2025Approximated Behavioral Metric-based State Projection for Federated Reinforcement Learningcited by 0
  6. 2026Constraint-Aware Aggregation for Federated Reinforcement Learning in Microgrid Energy Coordinationcited by 0