mlsanityData to 14 Oct 2026

FedGuide: Diffusion Prior Alignment and Value Baseline Guidance for Heterogeneous Federated Reinforcement Learning

arXiv 2609.18964 ↗First posted 22 Jul 2026cs.LG cs.ROcited by 0

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

Federated Reinforcement Learning (FRL) enables collaborative policy learning across distributed agents with heterogeneous environments. While recent methods based on variance reduction, divergence penalization, and momentum optimization improve FRL under heterogeneous settings, they still primarily synchronize policy or value-network parameters and do not explicitly address distributional mismatch among heterogeneous clients. Therefore, we propose FedGuide, a FRL framework that uses diffusion priors as behavior models to provide personalized data supported distributions for heterogeneous local policy learning. Instead of directly averaging local policies, FedGuide aggregates those diffusion priors through Optimal-Transport Mixture-of-Experts (OT-MoE), preserving heterogeneous behavior modes in distribution space. It further develops a Distribution Correction Estimation (DICE) value baseline to provide low-variance, return-aware guidance for local policy improvement. Experiments across heterogeneous environments show that FedGuide outperforms representative FRL methods in client-average returns, final-round performance, and worst-round robustness, while maintaining stable learning under stronger heterogeneity.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 41 of its 50 references that are in the map. First posted on arXiv 22 Jul 2026.

Builds on

41 of 50 references.

Closest ancestors · 6

Same direction, newest first

  1. 2026Motion Planning with Model-Based Diffusion via Constraint Optimization and Adaptive Scheduling16 Jul
  2. 2025Prior-Guided Diffusion Planning for Offline Reinforcement Learning16 May
  3. 2024Diffusion Policy Policy Optimization1 Sep
  4. 2024Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learning29 Jul
  5. 2024Policy-Guided Diffusion9 Apr
  6. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusion7 Mar

Other references · 35

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  3. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,316
  4. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  5. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  6. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  7. 2020Federated Learning with Matched Averagingcited by 564
  8. 2021Towards Personalized Federated Learningcited by 464
  9. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  10. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 188
  11. 2020GenDICE: Generalized Offline Estimation of Stationary Valuescited by 122
  12. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimationcited by 121
23 more
  1. 2021Federated Reinforcement Learning: Techniques, Applications, and Open Challengescited by 81
  2. 2019Lifelong Federated Reinforcement Learning: A Learning Architecture for Navigation in Cloud Robotic Systemscited by 78
  3. 2022COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimationcited by 71
  4. 2022Federated Reinforcement Learning with Environment Heterogeneitycited by 63
  5. 2021Fault-Tolerant Federated Reinforcement Learning with Theoretical Guaranteecited by 54
  6. 2023Dual RL: Unification and New Methods for Reinforcement and Imitation Learningcited by 48
  7. 2019Federated Transfer Reinforcement Learning for Autonomous Drivingcited by 44
  8. 2023The Blessing of Heterogeneity in Federated Q-Learning: Linear Speedup and Beyondcited by 36
  9. 2024Finite-Time Analysis of On-Policy Heterogeneous Federated Reinforcement Learningcited by 31
  10. 2022FedKL: Tackling Data Heterogeneity in Federated Reinforcement Learning by Penalizing KL Divergencecited by 29
  11. 2024FedMoE: Personalized Federated Learning via Heterogeneous Mixture of Expertscited by 27
  12. 2024pFedMoE: Data-Level Personalization with Mixture of Experts for Model-Heterogeneous Personalized Federated Learningcited by 24
  13. 2024ODICE: Revealing the Mystery of Distribution Correction Estimation via Orthogonal-gradient Updatecited by 21
  14. 2022Distributed Reinforcement Learning for Robot Teams: A Reviewcited by 16
  15. 2024Federated Offline Reinforcement Learning: Collaborative Single-Policy Coverage Sufficescited by 11
  16. 2026HFedMoE: Resource-aware Heterogeneous Federated Learning with Mixture-of-Expertscited by 11
  17. 2024Momentum for the Win: Collaborative Federated Reinforcement Learning across Heterogeneous Environmentscited by 8
  18. 2021A Survey of Distributed Optimization Methods for Multi-Robot Systemscited by 7
  19. 2023FedREP: A Byzantine-Robust, Communication-Efficient and Privacy-Preserving Framework for Federated Learningcited by 4
  20. 2025FedHPD: Heterogeneous Federated Reinforcement Learning via Policy Distillationcited by 4
  21. 2025Heterogeneous Federated Reinforcement Learning Using Wasserstein Barycenterscited by 4
  22. 2026Model-Based Diffusion Optimal Control for Multi-Robot Motion Planningcited by 2
  23. 2024Momentum-Based Federated Reinforcement Learning with Interaction and Communication Efficiencycited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Federated Reinforcement Learning in Heterogeneous Environments
  2. 2024CAESAR: Enhancing Federated RL in Heterogeneous MDPs through Convergence-Aware Sampling with Screening
  3. 2025Collaborative Value Function Estimation Under Model Mismatch: A Federated Temporal Difference Analysis
  4. 2024Federated Reinforcement Learning with Constraint Heterogeneity
  5. 2025On Global Convergence Rates for Federated Softmax Policy Gradient under Heterogeneous Environments
  6. 2026FedQHD: Closed-Form Function-Space Federated Reinforcement Learning
  7. 2026Federated Distributional Reinforcement Learning with Distributional Critic Regularization
  8. 2024Towards Fast Rates for Federated and Multi-Task Reinforcement Learning