# Risk-Averse Approximate Dynamic Programming with Quantile-Based Risk Measures

arXiv: https://arxiv.org/abs/1509.01920 · HTML: https://mlsanity.com/paper/1509.01920/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Sep 2015.

## Built on by (most cited first, 6 of 6)

- [Quantile-Based Deep Reinforcement Learning using Two-Timescale Policy Gradient Algorithms](https://mlsanity.com/paper/2305.07248/) (2023)
- [Risk-Averse Constrained Reinforcement Learning with Optimized Certainty Equivalents](https://mlsanity.com/paper/2510.20199/) (2025)
- [Reinforcement Learning for Exponential Utility: Algorithms and Convergence in Discounted MDPs](https://mlsanity.com/paper/2605.08053/) (2026)
- [Risk-Averse MDPs under Reward Ambiguity](https://mlsanity.com/paper/2301.01045/) (2023)
- [R2L: Reliable Reinforcement Learning: Guaranteed Return & Reliable Policies in Reinforcement Learning](https://mlsanity.com/paper/2510.18074/) (2025)
- [Long-Run Conditional Value-at-Risk Reinforcement Learning](https://mlsanity.com/paper/2603.09734/) (2026)

## Abstract (arXiv)

In this paper, we consider a finite-horizon Markov decision process (MDP) for which the objective at each stage is to minimize a quantile-based risk measure (QBRM) of the sequence of future costs; we call the overall objective a dynamic quantile-based risk measure (DQBRM). In particular, we consider optimizing dynamic risk measures where the one-step risk measures are QBRMs, a class of risk measures that includes the popular value at risk (VaR) and the conditional value at risk (CVaR). Although there is considerable theoretical development of risk-averse MDPs in the literature, the computational challenges have not been explored as thoroughly. We propose data-driven and simulation-based approximate dynamic programming (ADP) algorithms to solve the risk-averse sequential decision problem. We address the issue of inefficient sampling for risk applications in simulated settings and present a procedure, based on importance sampling, to direct samples toward the "risky region" as the ADP algorithm progresses. Finally, we show numerical results of our algorithms in the context of an application involving risk-averse bidding for energy storage.