mlsanityData to 7 Oct 2026

An Analysis of Categorical Distributional Reinforcement Learning

arXiv 1802.08163 ↗First posted 22 Feb 2018stat.MLcited by 68

Not on the map. Why?

Abstract (arXiv)

Distributional approaches to value-based reinforcement learning model the entire distribution of returns, rather than just their expected values, and have recently been shown to yield state-of-the-art empirical performance. This was demonstrated by the recently proposed C51 algorithm, based on categorical distributional reinforcement learning (CDRL) [Bellemare et al., 2017]. However, the theoretical properties of CDRL algorithms are not yet well understood. In this paper, we introduce a framework to analyse CDRL algorithms, establish the importance of the projected distributional Bellman operator in distributional RL, draw fundamental connections between CDRL and the Cram'er distance, and give a proof of convergence for sample-based categorical distributional reinforcement learning algorithms.

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 Feb 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 13; 2022: 7; 2023: 16; 2024: 11; 2025: 9; 2026: 12.

68 papers.

Sort:

By area:

  1. 2021On The Effect of Auxiliary Tasks on Representation Dynamicscited by 56
  2. 2021Universal Off-Policy Evaluationcited by 43
  3. 2021Exponential Bellman Equation and Improved Regret Bounds for Risk-Sensitive Reinforcement Learningcited by 42
  4. 2023An Analysis of Quantile Temporal-Difference Learningcited by 30
  5. 2023The Benefits of Being Distributional: Small-Loss Bounds for Reinforcement Learningcited by 27
  6. 2024More Benefits of Being Distributional: Second-Order Bounds for Reinforcement Learningcited by 23
  7. 2021Distributional Reinforcement Learning for Multi-Dimensional Reward Functionscited by 21
  8. 2023Trust Region-Based Safe Distributional Reinforcement Learning for Multiple Constraintscited by 20
  9. 2023Distributional Offline Policy Evaluation with Predictive Error Guaranteescited by 19
  10. 2021Unifying Cardiovascular Modelling with Deep Reinforcement Learning for Uncertainty Aware Control of Sepsis Treatmentcited by 16
  11. 2022Bridging Distributional and Risk-sensitive Reinforcement Learning with Provable Regret Boundscited by 16
  12. 2024Foundations of Multivariate Distributional Reinforcement Learningcited by 16
38 more
  1. 2021GMAC: A Distributional Perspective on Actor-Critic Frameworkcited by 15
  2. 2023The Statistical Benefits of Quantile Temporal-Difference Learning for Value Estimationcited by 14
  3. 2024Investigating the Histogram Loss in Regressioncited by 11
  4. 2022The Nature of Temporal Difference Errors in Multi-step Distributional Reinforcement Learningcited by 9
  5. 2023Distributional Model Equivalence for Risk-Sensitive Reinforcement Learningcited by 9
  6. 2023Diverse Projection Ensembles for Distributional Reinforcement Learningcited by 8
  7. 2023Estimation and Inference in Distributional Reinforcement Learningcited by 8
  8. 2021A Cram'er Distance perspective on Quantile Regression based Distributional Reinforcement Learningcited by 7
  9. 2022Reinforcement Learning with Heterogeneous Data: Estimation and Inferencecited by 7
  10. 2024Near-Minimax-Optimal Distributional Reinforcement Learning with a Generative Modelcited by 7
  11. 2021Enabling risk-aware Reinforcement Learning for medical interventions through uncertainty decompositioncited by 6
  12. 2022Revisiting Gaussian mixture critics in off-policy reinforcement learning: a sample-based approachcited by 6
  13. 2021Robustness and risk management via distributional dynamic programmingcited by 5
  14. 2023One-Step Distributional Reinforcement Learningcited by 5
  15. 2023Value-Distributional Model-Based Reinforcement Learningcited by 5
  16. 2024CTD4 -- A Deep Continuous Distributional Actor-Critic Agent with a Kalman Fusion of Multiple Criticscited by 5
  17. 2025A Finite Sample Analysis of Distributional TD Learning with Linear Function Approximationcited by 5
  18. 2025A Differential Perspective on Distributional Reinforcement Learningcited by 5
  19. 2021Intrinsic Benefits of Categorical Distributional Loss: Uncertainty-aware Regularized Exploration in Reinforcement Learningcited by 4
  20. 2023Exploration via Epistemic Value Estimationcited by 4
  21. 2023Distributional Bellman Operators over Mean Embeddingscited by 4
  22. 2021Bayesian Distributional Policy Gradientscited by 3
  23. 2021Distributional Reinforcement Learning with Unconstrained Monotonic Neural Networkscited by 3
  24. 2024Provable Risk-Sensitive Distributional Reinforcement Learning with General Function Approximationcited by 3
  25. 2021Conjugated Discrete Distributions for Distributional Reinforcement Learningcited by 2
  26. 2022On solutions of the distributional Bellman equationcited by 2
  27. 2022Improve Generalization of Driving Policy at Signalized Intersections with Adversarial Learningcited by 2
  28. 2023Distributional Multi-Objective Decision Makingcited by 2
  29. 2023PACER: A Fully Push-forward-based Distributional Reinforcement Learning Algorithmcited by 2
  30. 2023Variance Control for Distributional Reinforcement Learningcited by 2
  31. 2025Convergence Theorems for Entropy-Regularized and Distributional Reinforcement Learningcited by 2
  32. 2025Survey and Tutorial of Reinforcement Learning Methods in Process Systems Engineeringcited by 2
  33. 2022IGN : Implicit Generative Networkscited by 1
  34. 2024Off-policy Distributional Q(λ): Distributional RL without Importance Samplingcited by 1
  35. 2024On the Approximation and Convergence of Distributional Policy Gradient Algorithms for Risk-Sensitive Reinforcement Learningcited by 1
  36. 2025ADDQ: Adaptive Distributional Double Q-Learningcited by 1
  37. 2026A Spectral Revisit of the Distributional Bellman Operator under the Cram'er Metriccited by 1
  38. 2026Quotient-Categorical Representations for Bellman-Compatible Average-Reward Distributional Reinforcement Learningcited by 1

50 most cited of 68 All 68 (JSON)