mlsanityData to 14 Oct 2026

Multivariate Distributional Reinforcement Learning Using Sliced Divergences

arXiv 2605.31222 ↗First posted 29 May 2026cs.LGcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Distributional reinforcement learning (DRL) models the full return distribution rather than expectations, but extending it to multivariate settings remains challenging. Many common metrics do not naturally generalize beyond one dimension or lose computational tractability, and the multivariate case introduces additional difficulties such as general matrix discounting, for which no contraction results are available. We introduce Sliced Distributional Reinforcement Learning (SDRL), which lifts tractable one-dimensional divergences to multivariate return distributions via projections. We prove Bellman contraction for uniform slicing under shared scalar discounting, and introduce a maximum-slicing variant with contraction under general dense discount matrices. SDRL supports a broad class of base divergences; we analyze Wasserstein, Cram'er, and Maximum Mean Discrepancy (MMD), and characterize which SDRL variants suit the standard single-sample Bellman update used in distributional RL. We evaluate SDRL on a toy chain problem and a gridworld image-based environment as well as a subset of Atari games.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 12 of its 57 references that are in the map. First posted on arXiv 29 May 2026.

Builds on

12 of 57 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Foundations of Multivariate Distributional Reinforcement Learning31 Aug

Other references · 11

  1. 2015Continuous control with deep reinforcement learningcited by 3,407
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  3. 2018Demystifying MMD GANscited by 1,293
  4. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 772
  5. 2017The Cramer Distance as a Solution to Biased Wasserstein Gradientscited by 119
  6. 2019Learning with minibatch Wasserstein : asymptotic and gradient propertiescited by 87
  7. 2024Simplifying Deep Temporal Difference Learningcited by 78
  8. 2019Hyperbolic Discounting and Learning over Multiple Horizonscited by 60
  9. 2021Wasserstein GANs Work Because They Fail (to Approximate the Wasserstein Distance)cited by 35
  10. 2019How Well Do WGANs Estimate the Wasserstein Metric?cited by 17
  11. 2021A Cram'er Distance perspective on Quantile Regression based Distributional Reinforcement Learningcited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Near-Minimax-Optimal Distributional Reinforcement Learning with a Generative Model
  2. 2024Off-Policy Reinforcement Learning with High Dimensional Reward
  3. 2026Distributional value gradients for stochastic environments
  4. 2025A Differential Perspective on Distributional Reinforcement Learning
  5. 2026Stein Variational Distributional Reinforcement Learning: Score-Guided Particles for Multivariate Return Distributions
  6. 2024Bellman Unbiasedness: Toward Provably Efficient Distributional Reinforcement Learning with General Value Function Approximation
  7. 2024More Benefits of Being Distributional: Second-Order Bounds for Reinforcement Learning
  8. 2026Vector-Valued Distributional Reinforcement Learning Policy Evaluation: A Hilbert Space Embedding Approach