mlsanityData to 7 Oct 2026

Distributional Reinforcement Learning with Regularized Wasserstein Loss

arXiv 2202.00769 ↗First posted 1 Feb 2022cs.LG stat.MLcited by 8

Not on the map. Why?

Abstract (arXiv)

The empirical success of distributional reinforcement learning (RL) highly relies on the choice of distribution divergence equipped with an appropriate distribution representation. In this paper, we propose textitSinkhorn distributional RL (SinkhornDRL), which leverages Sinkhorn divergence, a regularized Wasserstein loss, to minimize the difference between current and target Bellman return distributions. Theoretically, we prove the contraction properties of SinkhornDRL, aligning with the interpolation nature of Sinkhorn divergence between Wasserstein distance and Maximum Mean Discrepancy (MMD). The introduced SinkhornDRL enriches the family of distributional RL algorithms, contributing to interpreting the algorithm behaviors compared with existing approaches by our investigation into their relationships. Empirically, we show that SinkhornDRL consistently outperforms or matches existing algorithms on the Atari games suite and particularly stands out in the multi-dimensional reward setting. thanksCode is available in urlhttps://github.com/datake/SinkhornDistRL..

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 27 of its 60 references that are in the map. First posted on arXiv 1 Feb 2022.

Builds on

27 of 60 references.

References · 27

  1. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,277
  2. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  3. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  4. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  5. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
  6. 2017Learning Generative Models with Sinkhorn Divergencescited by 310
  7. 2013A survey of the Schr"odinger problem and some of its connections with optimal transportcited by 286
  8. 2017Near-linear time approximation algorithms for optimal transport via Sinkhorn iterationcited by 219
  9. 2015On Wasserstein Two Sample Testing and Related Families of Nonparametric Testscited by 165
  10. 2021Unbalanced minibatch Optimal Transport; applications to Domain Adaptationcited by 123
  11. 2017The Cramer Distance as a Solution to Biased Wasserstein Gradientscited by 116
  12. 2019Wasserstein Adversarial Examples via Projected Sinkhorn Iterationscited by 89
15 more
  1. 2019Fully Parameterized Quantile Function for Distributional Reinforcement Learningcited by 87
  2. 2021Conservative Offline Distributional Reinforcement Learningcited by 78
  3. 2019Statistics and Samples in Distributional Reinforcement Learningcited by 62
  4. 2019Distributional Reinforcement Learning for Efficient Explorationcited by 59
  5. 2017Hybrid Reward Architecture for Reinforcement Learningcited by 57
  6. 2021Don't Generate Me: Training Differentially Private Generative Models with Sinkhorn Divergencecited by 54
  7. 2020Optimal Bounds between f-Divergences and Integral Probability Metricscited by 27
  8. 2021Quantitative Stability of Regularized Optimal Transport and Convergence of Sinkhorn's Algorithmcited by 21
  9. 2021Distributional Reinforcement Learning for Multi-Dimensional Reward Functionscited by 21
  10. 2019Screening Sinkhorn Algorithm for Regularized Optimal Transportcited by 17
  11. 2019Distributional Reward Decomposition for Reinforcement Learningcited by 11
  12. 2021Exploring the Training Robustness of Distributional Reinforcement Learning against Noisy State Observationscited by 11
  13. 2020Distributional Reinforcement Learning via Moment Matchingcited by 10
  14. 2021Intrinsic Benefits of Categorical Distributional Loss: Uncertainty-aware Regularized Exploration in Reinforcement Learningcited by 4
  15. 2020The energy distance for ensemble and scenario reductioncited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 0; 2025: 3; 2026: 4.

8 papers.

  1. 2025Qsharp: Provably Optimal Distributional RL for LLM Post-Trainingcited by 14
  2. 2025A Finite Sample Analysis of Distributional TD Learning with Linear Function Approximationcited by 5
  3. 2023PACER: A Fully Push-forward-based Distributional Reinforcement Learning Algorithmcited by 2
  4. 2026A Spectral Revisit of the Distributional Bellman Operator under the Cram'er Metriccited by 1
  5. 2025Distributionally Robust Control with End-to-End Statistically Guaranteed Metric Learningcited by 0
  6. 2026DFPO: Scaling Value Modeling via Distributional Flow towards Robust and Generalizable LLM Post-Trainingcited by 0
  7. 2026Learning the Supports for Categorical Critic in Reinforcement Learningcited by 0
  8. 2026Distributional Soft Bellman Operator under the Cram'er Geometrycited by 0