mlsanityData to 7 Oct 2026

Distributional Reinforcement Learning via the Cram'er Distance

arXiv 2605.08104 ↗First posted 26 Apr 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

This paper explores the application of the Soft Actor-Critic (SAC) algorithm within a Distributional Reinforcement Learning setting and introduces an implementation of such algorithm named Cram'er-based Distributional Soft Actor-Critic (C-DSAC). The novel approach employs distributional reinforcement learning to represent state-action values, and minimizes the squared Cram'er distance for learning the distribution. Empirical results across various robotic benchmarks indicate that our algorithm surpasses the performance of baseline SAC and contemporary distributional methods, with the performance advantage becoming increasingly pronounced in high-complexity environments. To explain the efficiency of the new approach, we conduct an analysis showing that its superior performance is partly due to textitconfidence-driven Q-value updates: High-variance target distributions (low confidence in target) lead to more conservative model updates, thereby attenuating the impact of overestimated values. This work deepens the understanding of distributional reinforcement learning, offering insights into the algorithmic mechanisms governing convergence and value estimation.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 22 of its 32 references that are in the map. First posted on arXiv 26 Apr 2026.

Builds on

22 of 32 references.

References · 22

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  6. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  7. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  8. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  9. 2017Deep Reinforcement Learning that Matterscited by 727
  10. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  11. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
  12. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 342
10 more
  1. 2020Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPOcited by 184
  2. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  3. 2020Maxmin Q-learning: Controlling the Estimation Bias of Q-learningcited by 117
  4. 2017The Cramer Distance as a Solution to Biased Wasserstein Gradientscited by 116
  5. 2019Fully Parameterized Quantile Function for Distributional Reinforcement Learningcited by 87
  6. 2020Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errorscited by 80
  7. 2023The Benefits of Being Distributional: Small-Loss Bounds for Reinforcement Learningcited by 27
  8. 2024More Benefits of Being Distributional: Second-Order Bounds for Reinforcement Learningcited by 23
  9. 2021GMAC: A Distributional Perspective on Actor-Critic Frameworkcited by 15
  10. 2021A Cram'er Distance perspective on Quantile Regression based Distributional Reinforcement Learningcited by 7

Built on by

Not cited yet.