# Quantile-Based Deep Reinforcement Learning using Two-Timescale Policy Gradient Algorithms

arXiv: https://arxiv.org/abs/2305.07248 · HTML: https://mlsanity.com/paper/2305.07248/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 66 references that are in the map.
- First posted on arXiv 12 May 2023.

## Builds on (most cited first, 17 of 17)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [Rainbow: Combining Improvements in Deep Reinforcement Learning](https://mlsanity.com/paper/1710.02298/) (2017)
- [Risk-Constrained Reinforcement Learning with Percentile Risk Criteria](https://mlsanity.com/paper/1512.01629/) (2015)
- [A Finite Time Analysis of Temporal Difference Learning With Linear Function Approximation](https://mlsanity.com/paper/1806.02450/) (2018)
- [Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularization](https://mlsanity.com/paper/2007.06558/) (2020)
- [Algorithms for CVaR Optimization in MDPs](https://mlsanity.com/paper/1406.3339/) (2014)
- [Cumulative Prospect Theory Meets Reinforcement Learning: Prediction and Control](https://mlsanity.com/paper/1506.02632/) (2015)
- [Scalable Reinforcement Learning for Multi-Agent Networked Systems](https://mlsanity.com/paper/1912.02906/) (2019)
- [Adaptive Discretization in Online Reinforcement Learning](https://mlsanity.com/paper/2110.15843/) (2021)
- [Reliable Off-policy Evaluation for Reinforcement Learning](https://mlsanity.com/paper/2011.04102/) (2020)
- [Quantile-Based Policy Optimization for Reinforcement Learning](https://mlsanity.com/paper/2201.11463/) (2022)
- [Risk-Averse Approximate Dynamic Programming with Quantile-Based Risk Measures](https://mlsanity.com/paper/1509.01920/) (2015)
- [Stochastic Optimal Control With Dynamic, Time-Consistent Risk Constraints](https://mlsanity.com/paper/1511.06980/) (2015)

## Built on by (most cited first, 4 of 4)

- [Dual-Agent Deep Reinforcement Learning for Dynamic Pricing and Replenishment](https://mlsanity.com/paper/2410.21109/) (2024)
- [Measures of Variability for Risk-averse Policy Gradient](https://mlsanity.com/paper/2504.11412/) (2025)
- [Stochastic Approximation Methods for Distortion Risk Measure Optimization](https://mlsanity.com/paper/2510.04563/) (2025)
- [Long-Run Conditional Value-at-Risk Reinforcement Learning](https://mlsanity.com/paper/2603.09734/) (2026)

## Abstract (arXiv)

Classical reinforcement learning (RL) aims to optimize the expected cumulative reward. In this work, we consider the RL setting where the goal is to optimize the quantile of the cumulative reward. We parameterize the policy controlling actions by neural networks, and propose a novel policy gradient algorithm called Quantile-Based Policy Optimization (QPO) and its variant Quantile-Based Proximal Policy Optimization (QPPO) for solving deep RL problems with quantile objectives. QPO uses two coupled iterations running at different timescales for simultaneously updating quantiles and policy parameters, whereas QPPO is an off-policy version of QPO that allows multiple updates of parameters during one simulation episode, leading to improved algorithm efficiency. Our numerical results indicate that the proposed algorithms outperform the existing baseline algorithms under the quantile criterion.