mlsanityData to 14 Oct 2026

Distributional Soft Bellman Operator under the Cram'er Geometry

arXiv 2607.17897 ↗First posted 20 Jul 2026cs.LGcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Distributional soft policy iteration (DSPI) provides an important framework for combining distributional reinforcement learning (DRL) with maximum-entropy control, in which the policy evaluation step is governed by a distributional soft Bellman operator acting on entropy-regularised returns. Theoretical analysis of such an evaluation step requires a probability metric under which Bellman updates can be controlled, typically by showing that the operator contracts the distance between any two candidate return-distribution estimates. In this paper, we focus on the Cram'er geometry, a cumulative distribution function (CDF)-based metric with an L2 structure, and study whether the fixed-policy distributional soft Bellman operator has this contraction property and hence a unique fixed point under this metric. Working directly on an admissible CDF field domain, we formulate the CDF-level distributional soft Bellman operator, prove that it is a √γ-contraction, and obtain the corresponding unique fixed point together with convergent iterative policy evaluation. The CDF formulation also shows that this finite-Cram'er-domain property follows from a uniform first-moment condition on the combined one-step reward entropy shift, rather than from separate uniform boundedness assumptions on the reward and entropy terms. We then transport the same evaluation problem to the spectral domain by conjugation, obtaining an equivalent Hilbert-space representation of the same decision process. Taken together, these results identify the Cram'er-geometric Bellman fixed point associated with the policy-evaluation step of DSPI, providing a reference point for studying approximate critics, evaluation error, and critic-loss design in DSPI-style algorithms.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 22 of its 43 references that are in the map. First posted on arXiv 20 Jul 2026.

Builds on

22 of 43 references.

Closest ancestors · 1

Same direction, newest first

  1. 2026A Spectral Revisit of the Distributional Bellman Operator under the Cram'er Metric13 Mar

Other references · 21

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  3. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  4. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  5. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
  6. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 369
  7. 2019A Theory of Regularized Markov Decision Processescited by 286
  8. 2018Distributed Distributional Deterministic Policy Gradientscited by 199
  9. 2017A unified view of entropy-regularized Markov decision processescited by 182
  10. 2018Learning to Walk via Deep Reinforcement Learningcited by 158
  11. 2019Fully Parameterized Quantile Function for Distributional Reinforcement Learningcited by 88
  12. 2020Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errorscited by 85
9 more
  1. 2018An Analysis of Categorical Distributional Reinforcement Learningcited by 68
  2. 2019Statistics and Samples in Distributional Reinforcement Learningcited by 63
  3. 2019A Comparative Analysis of Expected and Distributional Reinforcement Learningcited by 47
  4. 2023Distributional Soft Actor-Critic with Three Refinementscited by 25
  5. 2019Distributional Policy Optimization: An Alternative Approach for Continuous Controlcited by 22
  6. 2019Distributional reinforcement learning with linear function approximationcited by 16
  7. 2020DSAC: Distributional Soft Actor-Critic for Risk-Sensitive Reinforcement Learningcited by 8
  8. 2022Distributional Reinforcement Learning with Regularized Wasserstein Losscited by 8
  9. 2023Distributional Bellman Operators over Mean Embeddingscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Near-Minimax-Optimal Distributional Reinforcement Learning with a Generative Model
  2. 2026Distributional Reinforcement Learning via the Cram'er Distance
  3. 2024Bellman Unbiasedness: Toward Provably Efficient Distributional Reinforcement Learning with General Value Function Approximation
  4. 2026Natural Policy Gradient as Doubly Smoothed Policy Iteration: A Bellman-Operator Framework
  5. 2026Path-Coupled Bellman Flows for Distributional Reinforcement Learning
  6. 2025Parameter-Efficient Distributional RL via Normalizing Flows and a Geometry-Aware Cram'er Surrogate
  7. 2025Soft Fitted Q-Iteration without Bellman Completeness: Occupancy Reweighting and Temperature Annealing
  8. 2024Koopman-Assisted Reinforcement Learning