# Conditionally Elicitable Dynamic Risk Measures for Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/2206.14666 · HTML: https://mlsanity.com/paper/2206.14666/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 77 references that are in the map.
- First posted on arXiv 29 Jun 2022.

## Builds on (most cited first, 20 of 20)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [Cyclical Learning Rates for Training Neural Networks](https://mlsanity.com/paper/1506.01186/) (2015)
- [Risk-sensitive Reinforcement Learning](https://mlsanity.com/paper/1311.2097/) (2013)
- [Recent Advances in Reinforcement Learning in Finance](https://mlsanity.com/paper/2112.04553/) (2021)
- [Convergence of the Deep BSDE Method for Coupled FBSDEs](https://mlsanity.com/paper/1811.01165/) (2018)
- [Beyond expectation: Deep joint mean and quantile regression for spatio-temporal problems](https://mlsanity.com/paper/1808.08798/) (2018)
- [Reinforcement Learning with Dynamic Convex Risk Measures](https://mlsanity.com/paper/2112.13414/) (2021)
- [On the Convergence and Optimality of Policy Gradient for Markov Coherent Risk](https://mlsanity.com/paper/2103.02827/) (2021)
- [Entropic Risk Measure in Policy Search](https://mlsanity.com/paper/1906.09090/) (2019)
- [Markov Decision Processes with Recursive Risk Measures](https://mlsanity.com/paper/2010.07220/) (2020)
- [Robust Risk-Aware Reinforcement Learning](https://mlsanity.com/paper/2108.10403/) (2021)
- [Constrained Risk-Averse Markov Decision Processes](https://mlsanity.com/paper/2012.02423/) (2020)
- [Risk-Averse Learning by Temporal Difference Methods](https://mlsanity.com/paper/2003.00780/) (2020)
- [Minimizing Spectral Risk Measures Applied to Markov Decision Processes](https://mlsanity.com/paper/2012.04521/) (2020)
- [Sensitivity Measures Based on Scoring Functions](https://mlsanity.com/paper/2203.00460/) (2022)
- [Practical Risk Measures in Reinforcement Learning](https://mlsanity.com/paper/1908.08379/) (2019)
- [Deep Reinforcement Learning for Equal Risk Pricing and Hedging under Dynamic Expectile Risk Measures](https://mlsanity.com/paper/2109.04001/) (2021)
- [Deep Quantile and Deep Composite Model Regression](https://mlsanity.com/paper/2112.03075/) (2021)

## Built on by (most cited first, 9 of 9)

- [Robust Risk-Aware Option Hedging](https://mlsanity.com/paper/2303.15216/) (2023)
- [Regret Bounds for Risk-sensitive Reinforcement Learning with Lipschitz Dynamic Risk Measures](https://mlsanity.com/paper/2306.02399/) (2023)
- [FuNVol: A Multi-Asset Implied Volatility Market Simulator using Functional Principal Components and Neural SDEs](https://mlsanity.com/paper/2303.00859/) (2023)
- [Robust Reinforcement Learning with Dynamic Distortion Risk Measures](https://mlsanity.com/paper/2409.10096/) (2024)
- [Eliciting Risk Aversion with Inverse Reinforcement Learning via Interactive Questioning](https://mlsanity.com/paper/2308.08427/) (2023)
- [Catastrophic-risk-aware reinforcement learning with extreme-value-theory-based policy gradients](https://mlsanity.com/paper/2406.15612/) (2024)
- [Actor-Critic Algorithm for Dynamic Expectile and CVaR](https://mlsanity.com/paper/2605.07857/) (2026)
- [Kullback-Leibler Barycentre of Stochastic Processes](https://mlsanity.com/paper/2407.04860/) (2024)
- [Online Policy Evaluation for MDPs with Dynamic UBSR Measures](https://mlsanity.com/paper/2607.23030/) (2026)

## Abstract (arXiv)

We propose a novel framework to solve risk-sensitive reinforcement learning (RL) problems where the agent optimises time-consistent dynamic spectral risk measures. Based on the notion of conditional elicitability, our methodology constructs (strictly consistent) scoring functions that are used as penalizers in the estimation procedure. Our contribution is threefold: we (i) devise an efficient approach to estimate a class of dynamic spectral risk measures with deep neural networks, (ii) prove that these dynamic spectral risk measures may be approximated to any arbitrary accuracy using deep neural networks, and (iii) develop a risk-sensitive actor-critic algorithm that uses full episodes and does not require any additional nested transitions. We compare our conceptually improved reinforcement learning algorithm with the nested simulation approach and illustrate its performance in two settings: statistical arbitrage and portfolio allocation on both simulated and real data.