# Approximate Value Iteration for Risk-aware Markov Decision Processes

arXiv: https://arxiv.org/abs/1701.01290 · HTML: https://mlsanity.com/paper/1701.01290/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Jan 2017.

## Built on by (most cited first, 9 of 9)

- [Reinforcement Learning with Dynamic Convex Risk Measures](https://mlsanity.com/paper/2112.13414/) (2021)
- [Provably Efficient Iterated CVaR Reinforcement Learning with Function Approximation and Human Feedback](https://mlsanity.com/paper/2307.02842/) (2023)
- [Truncated Variance Reduced Value Iteration](https://mlsanity.com/paper/2405.12952/) (2024)
- [Provably Efficient CVaR RL in Low-rank MDPs](https://mlsanity.com/paper/2311.11965/) (2023)
- [Risk-Averse Constrained Reinforcement Learning with Optimized Certainty Equivalents](https://mlsanity.com/paper/2510.20199/) (2025)
- [Reinforcement Learning with Markov Risk Measures and Multipattern Risk Approximation](https://mlsanity.com/paper/2605.00654/) (2026)
- [Distributional Method for Risk Averse Reinforcement Learning](https://mlsanity.com/paper/2302.14109/) (2023)
- [Online Policy Evaluation for MDPs with Dynamic UBSR Measures](https://mlsanity.com/paper/2607.23030/) (2026)
- [Mini-Batch Risk-Averse Deep Q-Learning: A Robot Navigation Case Study](https://mlsanity.com/paper/2609.07998/) (2026)

## Abstract (arXiv)

We consider large-scale Markov decision processes (MDPs) with a risk measure of variability in cost, under the risk-aware MDPs paradigm. Previous studies showed that risk-aware MDPs, based on a minimax approach to handling risk, can be solved using dynamic programming for small to medium sized problems. However, due to the "curse of dimensionality", MDPs that model real-life problems are typically prohibitively large for such approaches. In this paper, we employ an approximate dynamic programming approach, and develop a family of simulation-based algorithms to approximately solve large-scale risk-aware MDPs. In parallel, we develop a unified convergence analysis technique to derive sample complexity bounds for this new family of algorithms.