FedQHD: Closed-Form Function-Space Federated Reinforcement Learning
Not on the map. Why?
Abstract (arXiv)
Federated reinforcement learning enables decentralized agents to collaboratively improve policies or value estimates without exchanging raw trajectories. However, FedAvg-style parameter averaging is not function-space consistent: when clients use heterogeneous encoders or even identical nonlinear networks, averaged parameters need not correspond to the weighted average of client value functions in any common function space. We propose FedQHD, a federated Q-learning method using hyperdimensional (random-feature) state encoders with a linear readout, so that Q-functions are nonlinear in state yet linear in trainable parameters. This linear structure enables closed-form aggregation. With a shared encoder, the function-space consensus update coincides exactly with weighted averaging of local readout matrices. With heterogeneous encoders, the server constructs a global teacher by averaging client Q-values on a shared anchor-state set, and each client compiles this teacher into its local representation via a single ridge projection. We formalize the federation gap -- the error incurred when compiling a federated teacher into a heterogeneous client representation -- relative to a client-specific oracle projection. We show that this gap decomposes into subspace misalignment, anchor-set conditioning, and regularization bias. We further identify the anchor-to-dimension ratio m ≥ Di as the well-conditioned regime in which the gap reduces to a multiple of the encoder heterogeneity floor. On four continuous-state, discrete-action control benchmarks, FedQHD matches or outperforms FedAvg-style baselines and distillation-based alternatives while requiring substantially less computation, and the empirical dependence of the federation gap on encoder dimension matches our theoretical analysis.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 20 of its 34 references that are in the map. First posted on arXiv 27 May 2026.
Builds on
20 of 34 references.
References · 20
- 2015Distilling the Knowledge in a Neural Networkcited by 10,076
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
- 2018Federated Optimization in Heterogeneous Networkscited by 2,971
- 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
- 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 514
- 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 454
- 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 380
- 2017Distral: Robust Multitask Reinforcement Learningcited by 231
- 2018A Finite Time Analysis of Temporal Difference Learning With Linear Function Approximationcited by 225
- 2020FedBE: Making Bayesian Model Ensemble Applicable to Federated Learningcited by 176
- 2016Generalization Properties of Learning with Random Featurescited by 163
- 2019Distilling Policy Distillationcited by 81
8 more
- 2021Federated Reinforcement Learning: Techniques, Applications, and Open Challengescited by 79
- 2019Lifelong Federated Reinforcement Learning: A Learning Architecture for Navigation in Cloud Robotic Systemscited by 77
- 2022Federated Reinforcement Learning with Environment Heterogeneitycited by 61
- 2021Fault-Tolerant Federated Reinforcement Learning with Theoretical Guaranteecited by 52
- 2019Federated Transfer Reinforcement Learning for Autonomous Drivingcited by 43
- 2023Federated Learning for Connected and Automated Vehicles: A Survey of Existing Approaches and Challengescited by 39
- 2020When Deep Reinforcement Learning Meets Federated Learning: Intelligent Multi-Timescale Resource Management for Multi-access Edge Computing in 5G Ultra Dense Networkcited by 29
- 2025FedHPD: Heterogeneous Federated Reinforcement Learning via Policy Distillationcited by 4
Built on by
Not cited yet.