mlsanityData to 14 Oct 2026

Risk-Averse Reinforcement Learning with Itakura-Saito Loss

arXiv 2505.16925 ↗First posted 22 May 2025cs.LGcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Risk-averse reinforcement learning finds application in various high-stakes fields. Unlike classical reinforcement learning, which aims to maximize expected returns, risk-averse agents choose policies that minimize risk, occasionally sacrificing expected value. These preferences can be framed through utility theory. We focus on the specific case of the exponential utility function, where one can derive the Bellman equations and employ various reinforcement learning algorithms with few modifications. To address this, we introduce to the broad machine learning community a numerically stable and mathematically sound loss function based on the Itakura-Saito divergence for learning state-value and action-value functions. We evaluate the Itakura-Saito loss function against established alternatives, both theoretically and empirically. In the experimental section, we explore multiple scenarios, some with known analytical solutions, and show that the considered loss function outperforms the alternatives.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 19 of its 62 references that are in the map. First posted on arXiv 22 May 2025.

Builds on

19 of 62 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Risk-Sensitive Soft Actor-Critic for Robust Deep Reinforcement Learning under Distribution Shifts15 Feb

Other references · 18

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  2. 2016Eigenvalues of the Hessian in Deep Learning: Singularity and Beyondcited by 159
  3. 2013Risk-sensitive Reinforcement Learningcited by 130
  4. 2021Recent Advances in Reinforcement Learning in Financecited by 110
  5. 2018Deep Hedgingcited by 108
  6. 2018Hessian-based Analysis of Large Batch Training and Robustness to Adversariescited by 87
  7. 2017A Random Matrix Approach to Neural Networkscited by 80
  8. 2018The Emergence of Spectral Universality in Deep Networkscited by 74
  9. 2020Risk-Sensitive Reinforcement Learning: Near-Optimal Risk-Sample Tradeoff in Regretcited by 58
  10. 2021Exponential Bellman Equation and Improved Regret Bounds for Risk-Sensitive Reinforcement Learningcited by 43
  11. 2021Deep Hedging of Derivatives Using Reinforcement Learningcited by 30
  12. 2017QLBS: Q-Learner in the Black-Scholes(-Merton) Worldscited by 23
6 more
  1. 2019Entropic Risk Measure in Policy Searchcited by 19
  2. 2022A Policy Gradient Algorithm for the Risk-Sensitive Exponential Cost MDPcited by 13
  3. 2022Deep Hedging: Continuous Reinforcement Learning for Hedging of General Portfolios across Multiple Risk Aversionscited by 13
  4. 2021Deep Hedging: Learning to Remove the Drift under Trading Frictions with Minimal Equivalent Near-Martingale Measurescited by 11
  5. 2021Model-Free Risk-Sensitive Reinforcement Learningcited by 7
  6. 2023Applying Reinforcement Learning to Option Pricing and Hedgingcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024The Central Role of the Loss Function in Reinforcement Learning
  2. 2025Risk-sensitive Reinforcement Learning Based on Convex Scoring Functions
  3. 2024Risk-averse Total-reward MDPs with ERM and EVaR
  4. 2025Risk-Averse Total-Reward Reinforcement Learning
  5. 2025Rectifying Regression in Reinforcement Learning
  6. 2025Optimistic Exploration for Risk-Averse Constrained Reinforcement Learning
  7. 2024Catastrophic-risk-aware reinforcement learning with extreme-value-theory-based policy gradients
  8. 2026Actor-Critic Algorithm for Dynamic Expectile and CVaR