Federated Control in Markov Decision Processes
Not on the map. Why?
Abstract (arXiv)
We study problems of federated control in Markov Decision Processes. To solve an MDP with large state space, multiple learning agents are introduced to collaboratively learn its optimal policy without communication of locally collected experience. In our settings, these agents have limited capabilities, which means they are restricted within different regions of the overall state space during the training process. In face of the difference among restricted regions, we firstly introduce concepts of leakage probabilities to understand how such heterogeneity affects the learning process, and then propose a novel communication protocol that we call Federated-Q protocol (FedQ), which periodically aggregates agents' knowledge of their restricted regions and accordingly modifies their learning problems for further training. In terms of theoretical analysis, we justify the correctness of FedQ as a communication protocol, then give a general result on sample complexity of derived algorithms FedQ-X with the RL oracle , and finally conduct a thorough study on the sample complexity of FedQ-SynQ. Specifically, FedQ-X has been shown to enjoy linear speedup in terms of sample complexity when workload is uniformly distributed among agents. Moreover, we carry out experiments in various environments to justify the efficiency of our methods.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 12 of its 27 references that are in the map. First posted on arXiv 7 May 2024.
Builds on
12 of 27 references.
References · 12
- 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 698
- 2019Model-Based Reinforcement Learning with a Generative Model is Minimax Optimalcited by 124
- 2020Breaking the Sample Size Barrier in Model-Based Reinforcement Learning with a Generative Modelcited by 108
- 2019Model-free Deep Reinforcement Learning for Urban Autonomous Drivingcited by 96
- 2019Reinforcement Learning to Optimize Long-term User Engagement in Recommender Systemscited by 90
- 2021Is Q-Learning Minimax Optimal? A Tight Sample Complexity Analysiscited by 89
- 2019Lifelong Federated Reinforcement Learning: A Learning Architecture for Navigation in Cloud Robotic Systemscited by 77
- 2019Variance-reduced Q-learning is minimax optimalcited by 72
- 2022Federated Reinforcement Learning with Environment Heterogeneitycited by 61
- 2023The Blessing of Heterogeneity in Federated Q-Learning: Linear Speedup and Beyondcited by 34
- 2019Model-Based Reinforcement Learning with Adversarial Training for Online Recommendationcited by 33
- 2015Optimal control in Markov decision processes via distributed optimizationcited by 1
Built on by
Not cited yet.