# Risk-Averse Reinforcement Learning via Dynamic Time-Consistent Risk Measures

arXiv: https://arxiv.org/abs/2301.05981 · HTML: https://mlsanity.com/paper/2301.05981/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 39 references that are in the map.
- First posted on arXiv 14 Jan 2023.

## Builds on (most cited first, 8 of 8)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Optimizing the CVaR via Sampling](https://mlsanity.com/paper/1404.3862/) (2014)
- [Algorithms for CVaR Optimization in MDPs](https://mlsanity.com/paper/1406.3339/) (2014)
- [Distributionally Robust Counterpart in Markov Decision Processes](https://mlsanity.com/paper/1501.07418/) (2015)
- [Distributionally Robust Partially Observable Markov Decision Process with Moment-based Ambiguity](https://mlsanity.com/paper/1906.05988/) (2019)
- [Stochastic Optimal Control With Dynamic, Time-Consistent Risk Constraints](https://mlsanity.com/paper/1511.06980/) (2015)
- [A Framework for Time-Consistent, Risk-Averse Model Predictive Control: Theory and Algorithms](https://mlsanity.com/paper/1511.06981/) (2015)

## Built on by (most cited first, 5 of 5)

- [On the Global Convergence of Risk-Averse Natural Policy Gradient Methods with Expected Conditional Risk Measures](https://mlsanity.com/paper/2301.10932/) (2023)
- [Root Cause Attribution of Delivery Risks via Causal Discovery with Reinforcement Learning](https://mlsanity.com/paper/2408.05860/) (2024)
- [Sequential Decision-Making under Uncertainty: A Robust MDPs review](https://mlsanity.com/paper/2404.00940/) (2024)
- [On the Approximation and Convergence of Distributional Policy Gradient Algorithms for Risk-Sensitive Reinforcement Learning](https://mlsanity.com/paper/2405.14749/) (2024)
- [Long-Term Sequential Decision Making under Risk](https://mlsanity.com/paper/2607.19914/) (2026)

## Abstract (arXiv)

Traditional reinforcement learning (RL) aims to maximize the expected total reward, while the risk of uncertain outcomes needs to be controlled to ensure reliable performance in a risk-averse setting. In this paper, we consider the problem of maximizing dynamic risk of a sequence of rewards in infinite-horizon Markov Decision Processes (MDPs). We adapt the Expected Conditional Risk Measures (ECRMs) to the infinite-horizon risk-averse MDP and prove its time consistency. Using a convex combination of expectation and conditional value-at-risk (CVaR) as a special one-step conditional risk measure, we reformulate the risk-averse MDP as a risk-neutral counterpart with augmented action space and manipulation on the immediate rewards. We further prove that the related Bellman operator is a contraction mapping, which guarantees the convergence of any value-based RL algorithms. Accordingly, we develop a risk-averse deep Q-learning framework, and our numerical studies based on two simple MDPs show that the risk-averse setting can reduce the variance and enhance robustness of the results.