mlsanityData to 14 Oct 2026

A Spectral Revisit of the Distributional Bellman Operator under the Cram'er Metric

arXiv 2603.12576 ↗First posted 13 Mar 2026cs.LGcited by 1

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Distributional reinforcement learning (DRL) studies the evolution of full return distributions under Bellman updates rather than focusing on expected values. A classical result is that the distributional Bellman operator is contractive under the Cram'er metric, which corresponds to an L2 geometry on differences of cumulative distribution functions (CDFs). While this contraction ensures stability of policy evaluation, existing analyses remain largely metric, focusing on contraction properties without elucidating the structural action of the Bellman update on distributions. In this work, we analyse distributional Bellman dynamics directly at the level of CDFs, treating the Cram'er geometry as the intrinsic analytical setting. At this level, the Bellman update acts affinely on CDFs and linearly on differences between CDFs, and its contraction property yields a uniform bound on this linear action. Building on this intrinsic formulation, we construct a family of regularised spectral Hilbert representations that realise the CDF-level geometry by exact conjugation, without modifying the underlying Bellman dynamics. The regularisation affects only the geometry and vanishes in the zero-regularisation limit, recovering the native Cram'er metric. This framework clarifies the operator structure underlying distributional Bellman updates and provides a foundation for further functional and operator-theoretic analyses in DRL.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 15 of its 42 references that are in the map. First posted on arXiv 13 Mar 2026.

Builds on

15 of 42 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Optimizing Return Distributions with Distributional Dynamic Programming22 Jan

Other references · 14

  1. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  2. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
  3. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 369
  4. 2018Distributed Distributional Deterministic Policy Gradientscited by 199
  5. 2019Fully Parameterized Quantile Function for Distributional Reinforcement Learningcited by 88
  6. 2020Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errorscited by 85
  7. 2018An Analysis of Categorical Distributional Reinforcement Learningcited by 68
  8. 2019Statistics and Samples in Distributional Reinforcement Learningcited by 63
  9. 2019A Comparative Analysis of Expected and Distributional Reinforcement Learningcited by 47
  10. 2023Distributional Soft Actor-Critic with Three Refinementscited by 25
  11. 2019Distributional Policy Optimization: An Alternative Approach for Continuous Controlcited by 22
  12. 2019Distributional reinforcement learning with linear function approximationcited by 16
2 more
  1. 2022Distributional Reinforcement Learning with Regularized Wasserstein Losscited by 8
  2. 2023Distributional Bellman Operators over Mean Embeddingscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Distributional Soft Bellman Operator under the Cram'er Geometrycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Path-Coupled Bellman Flows for Distributional Reinforcement Learning
  2. 2025Parameter-Efficient Distributional RL via Normalizing Flows and a Geometry-Aware Cram'er Surrogate
  3. 2024Bellman Unbiasedness: Toward Provably Efficient Distributional Reinforcement Learning with General Value Function Approximation
  4. 2025Spectral Bellman Method: Unifying Representation and Exploration in RL
  5. 2024Near-Minimax-Optimal Distributional Reinforcement Learning with a Generative Model
  6. 2024Off-Policy Reinforcement Learning with High Dimensional Reward
  7. 2024Bellman Diffusion: Generative Modeling as Learning a Linear Operator in the Distribution Space
  8. 2025Bellman operator convergence enhancements in reinforcement learning algorithms