mlsanityData to 14 Oct 2026

Heterogeneous Federated Reinforcement Learning Using Wasserstein Barycenters

arXiv 2506.15825 ↗First posted 18 Jun 2025cs.LGcited by 4

On the map: Deep reinforcement learning methods → RL for energy, networks and operations

Abstract (arXiv)

In this paper, we first propose a novel algorithm for model fusion that leverages Wasserstein barycenters in training a global Deep Neural Network (DNN) in a distributed architecture. To this end, we divide the dataset into equal parts that are fed to "agents" who have identical deep neural networks and train only over the dataset fed to them (known as the local dataset). After some training iterations, we perform an aggregation step where we combine the weight parameters of all neural networks using Wasserstein barycenters. These steps form the proposed algorithm referred to as FedWB. Moreover, we leverage the processes created in the first part of the paper to develop an algorithm to tackle Heterogeneous Federated Reinforcement Learning (HFRL). Our test experiment is the CartPole toy problem, where we vary the lengths of the poles to create heterogeneous environments. We train a deep Q-Network (DQN) in each environment to learn to control each cart, while occasionally performing a global aggregation step to generalize the local models; the end outcome is a global DQN that functions across all environments.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in RL for energy, networks and operations, one of 15 directions in Deep reinforcement learning methods. Builds on 9 of its 28 references that are in the map. First posted on arXiv 18 Jun 2025.

Builds on

9 of 28 references.

References · 9

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  2. 2017Deep reinforcement learning from human preferencescited by 3,912
  3. 2019Federated Learning with Differential Privacy: Algorithms and Performance Analysiscited by 536
  4. 2019Release Strategies and the Social Impacts of Language Modelscited by 513
  5. 2019Model Fusion via Optimal Transportcited by 219
  6. 2021Federated Reinforcement Learning: Techniques, Applications, and Open Challengescited by 81
  7. 2022Federated Reinforcement Learning with Environment Heterogeneitycited by 63
  8. 2021Multi-task Learning with Attention for End-to-end Autonomous Drivingcited by 49
  9. 2022Wasserstein Barycenter-based Model Fusion and Linear Mode Connectivity of Neural Networkscited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Can Optimal Transport Improve Federated Inverse Reinforcement Learning?cited by 1
  2. 2026Federated Distributional Reinforcement Learning with Distributional Critic Regularizationcited by 1
  3. 2026FedGuide: Diffusion Prior Alignment and Value Baseline Guidance for Heterogeneous Federated Reinforcement Learningcited by 0
  4. 2026Optimal Transport Meets Reinforcement Learning: A Surveycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Federated Reinforcement Learning in Heterogeneous Environments
  2. 2026FedQHD: Closed-Form Function-Space Federated Reinforcement Learning
  3. 2024Momentum for the Win: Collaborative Federated Reinforcement Learning across Heterogeneous Environments
  4. 2025FedHPD: Heterogeneous Federated Reinforcement Learning via Policy Distillation
  5. 2024Finite-Time Analysis of On-Policy Heterogeneous Federated Reinforcement Learning
  6. 2024Federated Reinforcement Learning with Constraint Heterogeneity
  7. 2024On the Convergence Rates of Federated Q-Learning across Heterogeneous Environments
  8. 2024Single-Loop Federated Actor-Critic across Heterogeneous Environments