# Federated Distributional Reinforcement Learning with Distributional Critic Regularization

arXiv: https://arxiv.org/abs/2603.17820 · HTML: https://mlsanity.com/paper/2603.17820/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in RL for energy, networks and operations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 22 of its 35 references that are in the map.
- First posted on arXiv 18 Mar 2026.

## Placed in

- Deep reinforcement learning methods / [RL for energy, networks and operations](https://mlsanity.com/deeprl/rl-for-energy-networks-and-operations/)

## Builds on (most cited first, 22 of 22)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017)
- [A Distributional Perspective on Reinforcement Learning](https://mlsanity.com/paper/1707.06887/) (2017)
- [Distributional Reinforcement Learning with Quantile Regression](https://mlsanity.com/paper/1710.10044/) (2017)
- [Implicit Quantile Networks for Distributional Reinforcement Learning](https://mlsanity.com/paper/1806.06923/) (2018)
- [Trust Region Policy Optimisation in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2109.11251/) (2021)
- [Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approach](https://mlsanity.com/paper/1506.02188/) (2015)
- [Optimizing the CVaR via Sampling](https://mlsanity.com/paper/1404.3862/) (2014)
- [Federated Reinforcement Learning: Techniques, Applications, and Open Challenges](https://mlsanity.com/paper/2108.11887/) (2021)
- [Federated Reinforcement Learning with Environment Heterogeneity](https://mlsanity.com/paper/2204.02634/) (2022)
- [Trust Region-Based Safe Distributional Reinforcement Learning for Multiple Constraints](https://mlsanity.com/paper/2301.10923/) (2023)
- [Risk-Aware Reinforcement Learning through Optimal Transport Theory](https://mlsanity.com/paper/2309.06239/) (2023)
- [Understanding Reward Ambiguity Through Optimal Transport Theory in Inverse Reinforcement Learning](https://mlsanity.com/paper/2310.12055/) (2023)
- [Federated Ensemble-Directed Offline Reinforcement Learning](https://mlsanity.com/paper/2305.03097/) (2023)
- [The Synergy Between Optimal Transport Theory and Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2401.10949/) (2024)
- [Wasserstein Actor-Critic: Directed Exploration via Optimism for Continuous-Actions Control](https://mlsanity.com/paper/2303.02378/) (2023)
- [Heterogeneous Federated Reinforcement Learning Using Wasserstein Barycenters](https://mlsanity.com/paper/2506.15825/) (2025)
- [A Safe Deep Reinforcement Learning Approach for Energy Efficient Federated Learning in Wireless Communication Networks](https://mlsanity.com/paper/2308.10664/) (2023)
- [Wasserstein-Barycenter Consensus for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2506.12497/) (2025)
- [Neural Collapse-Inspired Multi-Label Federated Learning under Label-Distribution Skew](https://mlsanity.com/paper/2509.12544/) (2025)

## Built on by (most cited first, 1 of 1)

- [Uncertainty-Driven Replay Memory for Reinforcement Learning](https://mlsanity.com/paper/2608.29860/) (2026)

## Abstract (arXiv)

Federated reinforcement learning typically aggregates value functions or policies by parameter averaging, which emphasizes expected return and can obscure statistical multimodality and tail behavior that matter in safety-critical settings. We formalize federated distributional reinforcement learning (FedDistRL), where clients parametrize quantile value function critics and federate these networks only. We also propose TR-FedDistRL, which builds a per client, risk-aware Wasserstein barycenter over a temporal buffer. This local barycenter provides a reference region to constrain the parameter averaged critic, ensuring necessary distributional information is not averaged out during the federation process. The distributional trust region is implemented as a shrink-squash step around this reference. Under fixed-policy evaluation, the feasibility map is nonexpansive and the update is contractive in a probe-set Wasserstein metric under evaluation. Experiments on a bandit, multi-agent gridworld, and continuous highway environment show reduced mean-smearing, improved safety proxies (catastrophe/accident rate), and lower critic/policy drift versus mean-oriented and non-federated baselines.