# Fully Parameterized Quantile Function for Distributional Reinforcement Learning

arXiv: https://arxiv.org/abs/1911.02140 · HTML: https://mlsanity.com/paper/1911.02140/

## Key facts

- Cited by 88 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Nov 2019.

## Built on by (most cited first, 30 of 88)

- [Conservative Offline Distributional Reinforcement Learning](https://mlsanity.com/paper/2107.06106/) (2021)
- [Exponential Bellman Equation and Improved Regret Bounds for Risk-Sensitive Reinforcement Learning](https://mlsanity.com/paper/2111.03947/) (2021)
- [Multi-agent Reinforcement Learning: A Comprehensive Survey](https://mlsanity.com/paper/2312.10256/) (2023)
- [An Analysis of Quantile Temporal-Difference Learning](https://mlsanity.com/paper/2301.04462/) (2023)
- [Distributional GFlowNets with Quantile Flows](https://mlsanity.com/paper/2302.05793/) (2023)
- [The Benefits of Being Distributional: Small-Loss Bounds for Reinforcement Learning](https://mlsanity.com/paper/2305.15703/) (2023)
- [DFAC Framework: Factorizing the Value Function via Quantile Mixture for Multi-Agent Distributional Q-Learning](https://mlsanity.com/paper/2102.07936/) (2021)
- [Distributional Soft Actor-Critic with Three Refinements](https://mlsanity.com/paper/2310.05858/) (2023)
- [More Benefits of Being Distributional: Second-Order Bounds for Reinforcement Learning](https://mlsanity.com/paper/2402.07198/) (2024)
- [Distributional Reinforcement Learning for Multi-Dimensional Reward Functions](https://mlsanity.com/paper/2110.13578/) (2021)
- [AdaptVision: Efficient Vision-Language Models via Adaptive Visual Acquisition](https://mlsanity.com/paper/2512.03794/) (2025)
- [Probabilistic Time Series Forecasting with Implicit Quantile Networks](https://mlsanity.com/paper/2107.03743/) (2021)
- [A Distributional Analogue to the Successor Representation](https://mlsanity.com/paper/2402.08530/) (2024)
- [Foundations of Multivariate Distributional Reinforcement Learning](https://mlsanity.com/paper/2409.00328/) (2024)
- [GMAC: A Distributional Perspective on Actor-Critic Framework](https://mlsanity.com/paper/2105.11366/) (2021)
- [Bridging Distributional and Risk-sensitive Reinforcement Learning with Provable Regret Bounds](https://mlsanity.com/paper/2210.14051/) (2022)
- [Risk-Conditioned Distributional Soft Actor-Critic for Risk-Sensitive Navigation](https://mlsanity.com/paper/2104.03111/) (2021)
- [Non-decreasing Quantile Function Network with Efficient Exploration for Distributional Reinforcement Learning](https://mlsanity.com/paper/2105.06696/) (2021)
- [The Statistical Benefits of Quantile Temporal-Difference Learning for Value Estimation](https://mlsanity.com/paper/2305.18388/) (2023)
- [Learning Risk-Aware Quadrupedal Locomotion using Distributional Reinforcement Learning](https://mlsanity.com/paper/2309.14246/) (2023)
- [Deep Non-Crossing Quantiles through the Partial Derivative](https://mlsanity.com/paper/2201.12848/) (2022)
- [Distributional Hamilton-Jacobi-Bellman Equations for Continuous-Time Reinforcement Learning](https://mlsanity.com/paper/2205.12184/) (2022)
- [UI-Voyager: A Self-Evolving GUI Agent Learning via Failed Experience](https://mlsanity.com/paper/2603.24533/) (2026)
- [Safe Distributional Reinforcement Learning](https://mlsanity.com/paper/2102.13446/) (2021)
- [The Nature of Temporal Difference Errors in Multi-step Distributional Reinforcement Learning](https://mlsanity.com/paper/2207.07570/) (2022)
- [Quantile Constrained Reinforcement Learning: A Reinforcement Learning Framework Constraining Outage Probability](https://mlsanity.com/paper/2211.15034/) (2022)
- [Measuring Progress in Deep Reinforcement Learning Sample Efficiency](https://mlsanity.com/paper/2102.04881/) (2021)
- [A Cram'er Distance perspective on Quantile Regression based Distributional Reinforcement Learning](https://mlsanity.com/paper/2110.00535/) (2021)
- [Distributional Reinforcement Learning with Regularized Wasserstein Loss](https://mlsanity.com/paper/2202.00769/) (2022)
- [Diverse Projection Ensembles for Distributional Reinforcement Learning](https://mlsanity.com/paper/2306.07124/) (2023)

## Abstract (arXiv)

Distributional Reinforcement Learning (RL) differs from traditional RL in that, rather than the expectation of total returns, it estimates distributions and has achieved state-of-the-art performance on Atari Games. The key challenge in practical distributional RL algorithms lies in how to parameterize estimated distributions so as to better approximate the true continuous distribution. Existing distributional RL algorithms parameterize either the probability side or the return value side of the distribution function, leaving the other side uniformly fixed as in C51, QR-DQN or randomly sampled as in IQN. In this paper, we propose fully parameterized quantile function that parameterizes both the quantile fraction axis (i.e., the x-axis) and the value axis (i.e., y-axis) for distributional RL. Our algorithm contains a fraction proposal network that generates a discrete set of quantile fractions and a quantile value network that gives corresponding quantile values. The two networks are jointly trained to find the best approximation of the true distribution. Experiments on 55 Atari Games show that our algorithm significantly outperforms existing distributional RL algorithms and creates a new record for the Atari Learning Environment for non-distributed agents.