mlsanityData to 7 Oct 2026

Statistics and Samples in Distributional Reinforcement Learning

arXiv 1902.08102 ↗First posted 21 Feb 2019stat.ML cs.LGcited by 62

Not on the map. Why?

Abstract (arXiv)

We present a unifying framework for designing and analysing distributional reinforcement learning (DRL) algorithms in terms of recursively estimating statistics of the return distribution. Our key insight is that DRL algorithms can be decomposed as the combination of some statistical estimator and a method for imputing a return distribution consistent with that set of statistics. With this new understanding, we are able to provide improved analyses of existing DRL algorithms as well as construct a new algorithm (EDRL) based upon estimation of the expectiles of the return distribution. We compare EDRL with existing methods on a variety of MDPs to illustrate concrete aspects of our analysis, and develop a deep RL variant of the algorithm, ER-DQN, which we evaluate on the Atari-57 suite of games.

Key facts. Cited by 62 papers in the map (data through 7 Oct 2026). First posted on arXiv 21 Feb 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 8; 2023: 20; 2024: 4; 2025: 7; 2026: 11.

62 papers.

Sort:

By area:

  1. 2021Conservative Offline Distributional Reinforcement Learningcited by 78
  2. 2021Tactical Optimism and Pessimism for Deep Reinforcement Learningcited by 55
  3. 2021Universal Off-Policy Evaluationcited by 43
  4. 2021Offline Reinforcement Learning with Value-based Episodic Memorycited by 36
  5. 2023An Analysis of Quantile Temporal-Difference Learningcited by 30
  6. 2023Distributional GFlowNets with Quantile Flowscited by 28
  7. 2021DFAC Framework: Factorizing the Value Function via Quantile Mixture for Multi-Agent Distributional Q-Learningcited by 26
  8. 2023Distributional Soft Actor-Critic with Three Refinementscited by 22
  9. 2023RiskQ: Risk-sensitive Multi-Agent Reinforcement Learning Value Factorizationcited by 19
  10. 2021SENTINEL: Taming Uncertainty with Ensemble-based Distributional Reinforcement Learningcited by 15
  11. 2021GMAC: A Distributional Perspective on Actor-Critic Frameworkcited by 15
  12. 2023The Statistical Benefits of Quantile Temporal-Difference Learning for Value Estimationcited by 14
38 more
  1. 2023Learning Risk-Aware Quadrupedal Locomotion using Distributional Reinforcement Learningcited by 12
  2. 2021Deep Reinforcement Learning Policies Learn Shared Adversarial Features Across MDPscited by 11
  3. 2023Beyond Average Return in Markov Decision Processescited by 11
  4. 2022Distributional Hamilton-Jacobi-Bellman Equations for Continuous-Time Reinforcement Learningcited by 9
  5. 2022The Nature of Temporal Difference Errors in Multi-step Distributional Reinforcement Learningcited by 9
  6. 2023Distributional Model Equivalence for Risk-Sensitive Reinforcement Learningcited by 9
  7. 2022Distributional Reinforcement Learning with Regularized Wasserstein Losscited by 8
  8. 2023Diverse Projection Ensembles for Distributional Reinforcement Learningcited by 8
  9. 2022Reinforcement Learning with Heterogeneous Data: Estimation and Inferencecited by 7
  10. 2022Distributional Reward Estimation for Effective Multi-Agent Deep Reinforcement Learningcited by 6
  11. 2023Learning Diverse Risk Preferences in Population-based Self-playcited by 6
  12. 2021Robustness and risk management via distributional dynamic programmingcited by 5
  13. 2023Constrained Reinforcement Learning using Distributional Representation for Trustworthy Quadrotor UAV Tracking Controlcited by 5
  14. 2023Value-Distributional Model-Based Reinforcement Learningcited by 5
  15. 2025A Finite Sample Analysis of Distributional TD Learning with Linear Function Approximationcited by 5
  16. 2021Intrinsic Benefits of Categorical Distributional Loss: Uncertainty-aware Regularized Exploration in Reinforcement Learningcited by 4
  17. 2023Distributional Bellman Operators over Mean Embeddingscited by 4
  18. 2025Efficient Risk-sensitive Planning via Entropic Risk Measurescited by 4
  19. 2021Bayesian Distributional Policy Gradientscited by 3
  20. 2021Distributional Reinforcement Learning with Unconstrained Monotonic Neural Networkscited by 3
  21. 2022A Simulation Environment and Reinforcement Learning Method for Waste Reductioncited by 3
  22. 2022Distributional Actor-Critic Ensemble for Uncertainty-Aware Continuous Controlcited by 3
  23. 2023Is Risk-Sensitive Reinforcement Learning Properly Resolved?cited by 2
  24. 2023Variance Control for Distributional Reinforcement Learningcited by 2
  25. 2025Convergence Theorems for Entropy-Regularized and Distributional Reinforcement Learningcited by 2
  26. 2026LAD: Learning Advantage Distribution for Reasoningcited by 2
  27. 2022Interpretable Stochastic Model Predictive Control using Distributional Reinforced Estimation for Quadrotor Tracking Systemscited by 1
  28. 2023Distributional Reinforcement Learning with Dual Expectile-Quantile Regressioncited by 1
  29. 2023A Unified Framework for Factorizing Distributional Value Functions for Multi-Agent Reinforcement Learningcited by 1
  30. 2023Cramer Type Distances for Learning Gaussian Mixture Models by Gradient Descentcited by 1
  31. 2024Bootstrapping Expectiles in Reinforcement Learningcited by 1
  32. 2025Tackling Uncertainties in Multi-Agent Reinforcement Learning through Integration of Agent Termination Dynamicscited by 1
  33. 2026A Spectral Revisit of the Distributional Bellman Operator under the Cram'er Metriccited by 1
  34. 2026Quotient-Categorical Representations for Bellman-Compatible Average-Reward Distributional Reinforcement Learningcited by 1
  35. 2026OrderGrad: Optimizing Beyond the Mean with Order-Statistic Policy Gradient Estimationcited by 1
  36. 2023DRL-ORA: Distributional Reinforcement Learning with Online Risk Adaptioncited by 0
  37. 2023An introduction to reinforcement learning for neurosciencecited by 0
  38. 2024A Case for Validation Buffer in Pessimistic Actor-Criticcited by 0

50 most cited of 62 All 62 (JSON)