Risk-sensitive Reinforcement Learning Based on Convex Scoring Functions
Not on the map. Why?
Abstract (arXiv)
We propose a reinforcement learning (RL) framework under a broad class of risk objectives, characterized by convex scoring functions. This class covers many common risk measures, such as variance, Expected Shortfall, entropic Value-at-Risk, and mean-risk utility. To resolve the time-inconsistency issue, we consider an augmented state space and an auxiliary variable and recast the problem as a two-state optimization problem. We propose a customized Actor-Critic algorithm and establish some theoretical approximation guarantees. A key theoretical contribution is that our results do not require the Markov decision process to be continuous. Additionally, we propose an auxiliary variable sampling method inspired by the alternating minimization algorithm, which is convergent under certain conditions. We validate our approach in simulation experiments with a financial application in statistical arbitrage trading, demonstrating the effectiveness of the algorithm.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 18 of its 47 references that are in the map. First posted on arXiv 7 May 2025.
Builds on
18 of 47 references.
References · 18
- 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 320
- 2015Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approachcited by 162
- 2021Recent Advances in Reinforcement Learning in Financecited by 102
- 2014Optimizing the CVaR via Samplingcited by 98
- 2020Risk-Sensitive Reinforcement Learning: Near-Optimal Risk-Sample Tradeoff in Regretcited by 57
- 2021Exponential Bellman Equation and Improved Regret Bounds for Risk-Sensitive Reinforcement Learningcited by 42
- 2020Mean-Variance Policy Iteration for Risk-Averse Reinforcement Learningcited by 28
- 2022Provably Efficient Risk-Sensitive Reinforcement Learning: Iterated CVaR and Worst Pathcited by 21
- 2021Reinforcement Learning with Dynamic Convex Risk Measurescited by 20
- 2020Markov Decision Processes with Recursive Risk Measurescited by 17
- 2021Robust Risk-Aware Reinforcement Learningcited by 17
- 2024A Reductions Approach to Risk-Sensitive Reinforcement Learning with Optimized Certainty Equivalentscited by 15
6 more
- 2022Sensitivity Measures Based on Scoring Functionscited by 6
- 2024Continuous-time Risk-sensitive Reinforcement Learning via Quadratic Variation Penaltycited by 5
- 2021ACReL: Adversarial Conditional value-at-risk Reinforcement Learningcited by 2
- 2024Elicitability and identifiability of tail risk measurescited by 2
- 2020Convolution Bounds on Quantile Aggregationcited by 1
- 2024Robust Elicitable Functionalscited by 1
Built on by
Citing papers per year: 2026: 2.
2 papers.