Data to 7 Oct 2026

Distributional Reinforcement Learning with Quantile Regression

arXiv 1710.10044 ↗First posted 27 Oct 2017cs.AI cs.LG stat.MLcited by 464

Not on the map. Why?

Cited by 62 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In reinforcement learning an agent interacts with the environment by taking actions and observing the next state and reward. When sampled probabilistically, these state transitions, rewards, and actions can all induce randomness in the observed long-term return. Traditionally, reinforcement learning algorithms average over this randomness to estimate the value function. In this paper, we build on recent work advocating a distributional approach to reinforcement learning in which the distribution over returns is modeled explicitly instead of only estimating the mean. That is, we examine methods of learning the value distribution instead of the value function. We give results that close a number of gaps between the theoretical and algorithmic results given by Bellemare, Dabney, and Munos (2017). First, we extend existing results to the approximate distribution setting. Second, we present a novel distributional reinforcement learning algorithm consistent with our theoretical formulation. Finally, we evaluate this new algorithm on the Atari 2600 games, observing that it significantly outperforms many of the recent improvements on DQN, including the related distributional algorithm C51.

Key facts. Cited by 464 papers in the map (data through 7 Oct 2026). First posted on arXiv 27 Oct 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 1; 2019: 2; 2020: 1; 2021: 78; 2022: 75; 2023: 87; 2024: 64; 2025: 70; 2026: 86.

464 papers.

Sort:

By area:

2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
2018Reward learning from human preferences and demonstrations in Ataricited by 314
2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 307
2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learningcited by 125
2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 124
2019Benchmarking Batch Deep Reinforcement Learning Algorithmscited by 108
2021Generalized Decision Transformer for Offline Hindsight Information Matchingcited by 105
2021Online and Offline Reinforcement Learning by Planning with a Learned Modelcited by 90
2021d3rlpy: An Offline Deep Reinforcement Learning Librarycited by 84

38 more

2021Conservative Offline Distributional Reinforcement Learningcited by 78
2023A Survey on Uncertainty Quantification Methods for Deep Learningcited by 78
2024BOND: Aligning LLMs with Best-of-N Distillationcited by 73
2024SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learningcited by 70
2023Recent Advances in Optimal Transport for Machine Learningcited by 66
2022Reincarnating Reinforcement Learning: Reusing Prior Computation to Accelerate Progresscited by 63
2021A Survey of Deep RL and IL for Autonomous Driving Policy Learningcited by 59
2024Quantile Regression for Distributional Reward Models in RLHFcited by 59
2021Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domaincited by 57
2021On The Effect of Auxiliary Tasks on Representation Dynamicscited by 56
2021Tactical Optimism and Pessimism for Deep Reinforcement Learningcited by 55
2021The Difficulty of Passive Learning in Deep Reinforcement Learningcited by 50
2022A Survey on Quantum Reinforcement Learningcited by 50
2022Invariance in Policy Optimisation and Partial Identifiability in Reward Learningcited by 46
2021Universal Off-Policy Evaluationcited by 43
2021MICo: Improved representations via sampling-based state similarity for Markov decision processescited by 43
2021Exponential Bellman Equation and Improved Regret Bounds for Risk-Sensitive Reinforcement Learningcited by 42
2022Efficient Risk-Averse Reinforcement Learningcited by 41
2024Vision-Language Models Provide Promptable Representations for Reinforcement Learningcited by 41
2019An Optimistic Perspective on Offline Reinforcement Learningcited by 37
2021On Bonus-Based Exploration Methods in the Arcade Learning Environmentcited by 37
2021Offline Reinforcement Learning with Value-based Episodic Memorycited by 36
2022A Review of Uncertainty for Deep Reinforcement Learningcited by 36
2023Small batch deep reinforcement learningcited by 35
2021RMIX: Learning Risk-Sensitive Policies for Cooperative Reinforcement Learning Agentscited by 31
2023An Analysis of Quantile Temporal-Difference Learningcited by 30
2023On the Importance of Exploration for Generalization in Reinforcement Learningcited by 30
2023Multi-agent Reinforcement Learning: A Comprehensive Surveycited by 30
2021Off-Policy Risk Assessment in Contextual Banditscited by 29
2023Provable and Practical: Efficient Exploration in Reinforcement Learning via Langevin Monte Carlocited by 29
2023Distributional GFlowNets with Quantile Flowscited by 28
2021STEP: Stochastic Traversability Evaluation and Planning for Risk-Aware Off-road Navigationcited by 27
2021StARformer: Transformer with State-Action-Reward Representations for Visual Reinforcement Learningcited by 27
2021A Dataset Perspective on Offline Reinforcement Learningcited by 27
2022MuZero with Self-competition for Rate Control in VP9 Video Compressioncited by 27
2023The Benefits of Being Distributional: Small-Loss Bounds for Reinforcement Learningcited by 27
2021DFAC Framework: Factorizing the Value Function via Quantile Mixture for Multi-Agent Distributional Q-Learningcited by 26
2022Regret Bounds for Risk-Sensitive Reinforcement Learningcited by 26

50 most cited of 464 All 464 (JSON)