mlsanityData to 7 Oct 2026

A Distributional Perspective on Reinforcement Learning

arXiv 1707.06887 ↗First posted 21 Jul 2017cs.LG cs.AI stat.MLcited by 743

Not on the map. Why?

Cited by 69 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this paper we argue for the fundamental importance of the value distribution: the distribution of the random return received by a reinforcement learning agent. This is in contrast to the common approach to reinforcement learning which models the expectation of this return, or value. Although there is an established body of literature studying the value distribution, thus far it has always been used for a specific purpose such as implementing risk-aware behaviour. We begin with theoretical results in both the policy evaluation and control settings, exposing a significant distributional instability in the latter. We then use the distributional perspective to design a new algorithm which applies Bellman's equation to the learning of approximate value distributions. We evaluate our algorithm using the suite of games from the Arcade Learning Environment. We obtain both state-of-the-art results and anecdotal evidence demonstrating the importance of the value distribution in approximate reinforcement learning. Finally, we combine theoretical and empirical evidence to highlight the ways in which the value distribution impacts learning in the approximate setting.

Key facts. Cited by 743 papers in the map (data through 7 Oct 2026). First posted on arXiv 21 Jul 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 2; 2019: 5; 2020: 6; 2021: 123; 2022: 116; 2023: 121; 2024: 115; 2025: 118; 2026: 137.

743 papers.

Sort:

By area:

  1. 2023Mastering Diverse Domains through World Modelscited by 1,193
  2. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  3. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 482
  4. 2021Transformers Can Do Bayesian Inferencecited by 338
  5. 2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 337
  6. 2018Reward learning from human preferences and demonstrations in Ataricited by 314
  7. 2021GFlowNet Foundationscited by 286
  8. 2025π*0.6: a VLA That Learns From Experiencecited by 280
  9. 2022Multi-Game Decision Transformerscited by 212
  10. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 145
  11. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 136
  12. 2020Primal Wasserstein Imitation Learningcited by 120
38 more
  1. 2023Loss of Plasticity in Continual Deep Reinforcement Learningcited by 118
  2. 2020Hyperparameter Selection for Offline Reinforcement Learningcited by 113
  3. 2023Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learningcited by 111
  4. 2019Benchmarking Batch Deep Reinforcement Learning Algorithmscited by 108
  5. 2021Pretraining Representations for Data-Efficient Reinforcement Learningcited by 105
  6. 2021Generalized Decision Transformer for Offline Hindsight Information Matchingcited by 105
  7. 2024Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous controlcited by 104
  8. 2024Disentangling the Causes of Plasticity Loss in Neural Networkscited by 82
  9. 2021Conservative Offline Distributional Reinforcement Learningcited by 78
  10. 2023A Survey on Uncertainty Quantification Methods for Deep Learningcited by 78
  11. 2022Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizescited by 72
  12. 2021On the Expressivity of Markov Rewardcited by 71
  13. 2023Deep Reinforcement Learning with Plasticity Injectioncited by 71
  14. 2021What Matters for Adversarial Imitation Learning?cited by 70
  15. 2021Beyond Pick-and-Place: Tackling Robotic Stacking of Diverse Shapescited by 70
  16. 2023PFNs4BO: In-Context Learning for Bayesian Optimizationcited by 70
  17. 2022Automated Reinforcement Learning (AutoRL): A Survey and Open Problemscited by 69
  18. 2023Recent Advances in Optimal Transport for Machine Learningcited by 66
  19. 2021Risk-Averse Offline Reinforcement Learningcited by 65
  20. 2021Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspectivecited by 64
  21. 2021A Survey of Deep RL and IL for Autonomous Driving Policy Learningcited by 59
  22. 2023TorchRL: A data-driven decision-making library for PyTorchcited by 59
  23. 2024Quantile Regression for Distributional Reward Models in RLHFcited by 59
  24. 2021Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domaincited by 57
  25. 2021On The Effect of Auxiliary Tasks on Representation Dynamicscited by 56
  26. 2021Tactical Optimism and Pessimism for Deep Reinforcement Learningcited by 55
  27. 2022Offline RL Policies Should be Trained to be Adaptivecited by 52
  28. 2024Bridging State and History Representations: Understanding Self-Predictive RLcited by 52
  29. 2025DIME:Diffusion-Based Maximum Entropy Reinforcement Learningcited by 52
  30. 2026Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisonscited by 51
  31. 2022A-NeSI: A Scalable Approximate Method for Probabilistic Neurosymbolic Inferencecited by 50
  32. 2023SafeDreamer: Safe Reinforcement Learning with World Modelscited by 50
  33. 2024Learning Multimodal Behaviors from Scratch with Diffusion Policy Gradientcited by 49
  34. 2019Task-Relevant Adversarial Imitation Learningcited by 45
  35. 2021Universal Off-Policy Evaluationcited by 43
  36. 2021MICo: Improved representations via sampling-based state similarity for Markov decision processescited by 43
  37. 2021Exponential Bellman Equation and Improved Regret Bounds for Risk-Sensitive Reinforcement Learningcited by 42
  38. 2023PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learningcited by 42

50 most cited of 743 All 743 (JSON)