mlsanityData to 14 Oct 2026

Policy Newton methods for Distortion Riskmetrics

arXiv 2508.07249 ↗First posted 10 Aug 2025cs.LGcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

We consider the problem of risk-sensitive control in a reinforcement learning (RL) framework. In particular, we aim to find a risk-optimal policy by maximizing the distortion riskmetric (DRM) of the discounted reward in a finite horizon Markov decision process (MDP). DRMs are a rich class of risk measures that include several well-known risk measures as special cases. We derive a policy Hessian theorem for the DRM objective using the likelihood ratio method. Using this result, we propose a natural DRM Hessian estimator from sample trajectories of the underlying MDP. Next, we present a cubic-regularized policy Newton algorithm for solving this problem in an on-policy RL setting using estimates of the DRM gradient and Hessian. Our proposed algorithm is shown to converge to an ε-second-order stationary point (ε-SOSP) of the DRM objective, and this guarantee ensures the escaping of saddle points. The sample complexity of our algorithms to find an ε-SOSP is O(ε-3.5). Our experiments validate the theoretical findings. To the best of our knowledge, our is the first work to present convergence to an ε-SOSP of a risk-sensitive objective, while existing works in the literature have either shown convergence to a first-order stationary point of a risk-sensitive objective, or a SOSP of a risk-neutral one.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 13 of its 38 references that are in the map. First posted on arXiv 10 Aug 2025.

Builds on

13 of 38 references.

References · 13

  1. 2019Global Convergence of Policy Gradient Methods to (Almost) Locally Optimal Policiescited by 124
  2. 2015Policy Gradient for Coherent Risk Measurescited by 76
  3. 2015Cumulative Prospect Theory Meets Reinforcement Learning: Prediction and Controlcited by 34
  4. 2016Efficient approaches for escaping higher order saddle points in non-convex optimizationcited by 23
  5. 2020Sample Complexity of Policy Gradient Finding Second-Order Stationary Pointscited by 17
  6. 2023An Alternative to Variance: Gini Deviation for Risk-averse Policy Gradientcited by 12
  7. 2022Risk Measures and Upper Probabilities: Coherence and Stratificationcited by 9
  8. 2020Variational Model-based Policy Optimizationcited by 7
  9. 2023A Cubic-regularized Policy Newton Algorithm for Reinforcement Learningcited by 7
  10. 2021Policy Gradient Methods for Distortion Risk Measurescited by 6
  11. 2022A Risk-Sensitive Approach to Policy Optimizationcited by 6
  12. 2015A variational formula for risk-sensitive rewardcited by 3
1 more
  1. 2022Choquet regularization for reinforcement learningcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Policy Gradient for Robust Markov Decision Processes
  2. 2025Robust Bayesian Dynamic Programming for On-policy Risk-sensitive Reinforcement Learning
  3. 2025A Variance-Reduced Cubic-Regularized Newton for Policy Optimization
  4. 2024Pessimism Meets Risk: Risk-Sensitive Offline Reinforcement Learning
  5. 2025Bayesian Risk-Sensitive Policy Optimization For MDPs With General Loss Functions
  6. 2026Near-Optimal Sample Complexity for Online Constrained MDPs
  7. 2024Towards Efficient Risk-Sensitive Policy Gradient: An Iteration Complexity Analysis
  8. 2026Mathematical methods of reinforcement learning