# Markov Decision Processes with Recursive Risk Measures

arXiv: https://arxiv.org/abs/2010.07220 · HTML: https://mlsanity.com/paper/2010.07220/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Oct 2020.

## Built on by (most cited first, 18 of 18)

- [Provably Efficient Risk-Sensitive Reinforcement Learning: Iterated CVaR and Worst Path](https://mlsanity.com/paper/2206.02678/) (2022)
- [Reinforcement Learning with Dynamic Convex Risk Measures](https://mlsanity.com/paper/2112.13414/) (2021)
- [Soft Robust MDPs and Risk-Sensitive MDPs: Equivalence, Policy Gradient, and Sample Complexity](https://mlsanity.com/paper/2306.11626/) (2023)
- [Regret Bounds for Markov Decision Processes with Recursive Optimized Certainty Equivalents](https://mlsanity.com/paper/2301.12601/) (2023)
- [Risk-averse autonomous systems: A brief history and recent developments from the perspective of optimal control](https://mlsanity.com/paper/2109.08947/) (2021)
- [Conditionally Elicitable Dynamic Risk Measures for Deep Reinforcement Learning](https://mlsanity.com/paper/2206.14666/) (2022)
- [Regret Bounds for Risk-sensitive Reinforcement Learning with Lipschitz Dynamic Risk Measures](https://mlsanity.com/paper/2306.02399/) (2023)
- [Efficient Risk-sensitive Planning via Entropic Risk Measures](https://mlsanity.com/paper/2502.20423/) (2025)
- [Near-Optimal Sample Complexity for Iterated CVaR Reinforcement Learning with a Generative Model](https://mlsanity.com/paper/2503.08934/) (2025)
- [Risk-averse Total-reward MDPs with ERM and EVaR](https://mlsanity.com/paper/2408.17286/) (2024)
- [Robust Reinforcement Learning with Dynamic Distortion Risk Measures](https://mlsanity.com/paper/2409.10096/) (2024)
- [Is Risk-Sensitive Reinforcement Learning Properly Resolved?](https://mlsanity.com/paper/2307.00547/) (2023)
- [Risk-sensitive Reinforcement Learning Based on Convex Scoring Functions](https://mlsanity.com/paper/2505.04553/) (2025)
- [Reinforcement Learning with Markov Risk Measures and Multipattern Risk Approximation](https://mlsanity.com/paper/2605.00654/) (2026)
- [On the Sample Complexity of Discounted Reinforcement Learning with Optimized Certainty Equivalents](https://mlsanity.com/paper/2605.21763/) (2026)
- [Risk-Averse Total-Reward Reinforcement Learning](https://mlsanity.com/paper/2506.21683/) (2025)
- [Neural Architecture Search for global multi-step Forecasting of Energy Production Time Series](https://mlsanity.com/paper/2511.00035/) (2025)
- [Near-Optimal Sample Complexity for Recursive Entropic Risk Reinforcement Learning with a Generative Model](https://mlsanity.com/paper/2610.06931/) (2026)

## Abstract (arXiv)

In this paper, we consider risk-sensitive Markov Decision Processes (MDPs) with Borel state and action spaces and unbounded cost under both finite and infinite planning horizons. Our optimality criterion is based on the recursive application of static risk measures. This is motivated by recursive utilities in the economic literature, has been studied before for the entropic risk measure and is extended here to an axiomatic characterization of suitable risk measures. We derive a Bellman equation and prove the existence of Markovian optimal policies. For an infinite planning horizon, the model is shown to be contractive and the optimal policy to be stationary. Moreover, we establish a connection to distributionally robust MDPs, which provides a global interpretation of the recursively defined objective function. Monotone models are studied in particular.