# Stochastic Approximation for Risk-aware Markov Decision Processes

arXiv: https://arxiv.org/abs/1805.04238 · HTML: https://mlsanity.com/paper/1805.04238/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 May 2018.

## Built on by (most cited first, 10 of 10)

- [Exponential Bellman Equation and Improved Regret Bounds for Risk-Sensitive Reinforcement Learning](https://mlsanity.com/paper/2111.03947/) (2021)
- [Risk-Aware Transfer in Reinforcement Learning using Successor Features](https://mlsanity.com/paper/2105.14127/) (2021)
- [Risk-averse autonomous systems: A brief history and recent developments from the perspective of optimal control](https://mlsanity.com/paper/2109.08947/) (2021)
- [Risk-sensitive Markov Decision Process and Learning under General Utility Functions](https://mlsanity.com/paper/2311.13589/) (2023)
- [Risk-Averse Decision Making Under Uncertainty](https://mlsanity.com/paper/2109.04082/) (2021)
- [Distributional Method for Risk Averse Reinforcement Learning](https://mlsanity.com/paper/2302.14109/) (2023)
- [Risk-Sensitive Exponential Actor Critic](https://mlsanity.com/paper/2602.07202/) (2026)
- [Reinforcement Learning for Control with Probabilistic Stability Guarantee: A Finite-Sample Approach](https://mlsanity.com/paper/2603.00043/) (2026)
- [Finite-Time Analysis of Discounted Exponential-Utility Reinforcement Learning](https://mlsanity.com/paper/2608.01917/) (2026)
- [Adaptive Finite-Budget Training for CVaR Risk-Aware Q-Learning](https://mlsanity.com/paper/2608.04305/) (2026)

## Abstract (arXiv)

We develop a stochastic approximation-type algorithm to solve finite state/action, infinite-horizon, risk-aware Markov decision processes. Our algorithm has two loops. The inner loop computes the risk by solving a stochastic saddle-point problem. The outer loop performs Q-learning to compute an optimal risk-aware policy. Several widely investigated risk measures (e.g. conditional value-at-risk, optimized certainty equivalent, and absolute semi-deviation) are covered by our algorithm. Almost sure convergence and the convergence rate of the algorithm are established. For an error tolerance ε>0 for the optimal Q-value estimation gap and learning rate k∈(1/2,,1], the overall convergence rate of our algorithm is Omega((ln(1/δε)/ε2)1/k+(ln(1/ε))1/(1-k)) with probability at least 1-δ.