# Distributional Method for Risk Averse Reinforcement Learning

arXiv: https://arxiv.org/abs/2302.14109 · HTML: https://mlsanity.com/paper/2302.14109/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 31 references that are in the map.
- First posted on arXiv 27 Feb 2023.

## Builds on (most cited first, 11 of 11)

- [A Distributional Perspective on Reinforcement Learning](https://mlsanity.com/paper/1707.06887/) (2017)
- [Implicit Quantile Networks for Distributional Reinforcement Learning](https://mlsanity.com/paper/1806.06923/) (2018)
- [Risk-sensitive Reinforcement Learning](https://mlsanity.com/paper/1311.2097/) (2013)
- [Fully Parameterized Quantile Function for Distributional Reinforcement Learning](https://mlsanity.com/paper/1911.02140/) (2019)
- [Policy Gradient for Coherent Risk Measures](https://mlsanity.com/paper/1502.03919/) (2015)
- [On the Convergence and Optimality of Policy Gradient for Markov Coherent Risk](https://mlsanity.com/paper/2103.02827/) (2021)
- [Non-decreasing Quantile Function Network with Efficient Exploration for Distributional Reinforcement Learning](https://mlsanity.com/paper/2105.06696/) (2021)
- [Risk-Averse Learning by Temporal Difference Methods](https://mlsanity.com/paper/2003.00780/) (2020)
- [Stochastic Approximation for Risk-aware Markov Decision Processes](https://mlsanity.com/paper/1805.04238/) (2018)
- [Approximate Value Iteration for Risk-aware Markov Decision Processes](https://mlsanity.com/paper/1701.01290/) (2017)
- [Risk-sensitive Markov decision problems under model uncertainty: finite time horizon case](https://mlsanity.com/paper/2104.06915/) (2021)

## Abstract (arXiv)

We introduce a distributional method for learning the optimal policy in risk averse Markov decision process with finite state action spaces, latent costs, and stationary dynamics. We assume sequential observations of states, actions, and costs and assess the performance of a policy using dynamic risk measures constructed from nested Kusuoka-type conditional risk mappings. For such performance criteria, randomized policies may outperform deterministic policies, therefore, the candidate policies lie in the d-dimensional simplex where d is the cardinality of the action space. Existing risk averse reinforcement learning methods seldom concern randomized policies, na"ive extensions to current setting suffer from the curse of dimensionality. By exploiting certain structures embedded in the corresponding dynamic programming principle, we propose a distributional learning method for seeking the optimal policy. The conditional distribution of the value function is casted into a specific type of function, which is chosen with in mind the ease of risk averse optimization. We use a deep neural network to approximate said function, illustrate that the proposed method avoids the curse of dimensionality in the exploration phase, and explore the method's performance with a wide range of model parameters that are picked randomly.