# Policy Newton methods for Distortion Riskmetrics

arXiv: https://arxiv.org/abs/2508.07249 · HTML: https://mlsanity.com/paper/2508.07249/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 13 of its 38 references that are in the map.
- First posted on arXiv 10 Aug 2025.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 13 of 13)

- [Global Convergence of Policy Gradient Methods to (Almost) Locally Optimal Policies](https://mlsanity.com/paper/1906.08383/) (2019)
- [Policy Gradient for Coherent Risk Measures](https://mlsanity.com/paper/1502.03919/) (2015)
- [Cumulative Prospect Theory Meets Reinforcement Learning: Prediction and Control](https://mlsanity.com/paper/1506.02632/) (2015)
- [Efficient approaches for escaping higher order saddle points in non-convex optimization](https://mlsanity.com/paper/1602.05908/) (2016)
- [Sample Complexity of Policy Gradient Finding Second-Order Stationary Points](https://mlsanity.com/paper/2012.01491/) (2020)
- [An Alternative to Variance: Gini Deviation for Risk-averse Policy Gradient](https://mlsanity.com/paper/2307.08873/) (2023)
- [Risk Measures and Upper Probabilities: Coherence and Stratification](https://mlsanity.com/paper/2206.03183/) (2022)
- [Variational Model-based Policy Optimization](https://mlsanity.com/paper/2006.05443/) (2020)
- [A Cubic-regularized Policy Newton Algorithm for Reinforcement Learning](https://mlsanity.com/paper/2304.10951/) (2023)
- [Policy Gradient Methods for Distortion Risk Measures](https://mlsanity.com/paper/2107.04422/) (2021)
- [A Risk-Sensitive Approach to Policy Optimization](https://mlsanity.com/paper/2208.09106/) (2022)
- [A variational formula for risk-sensitive reward](https://mlsanity.com/paper/1501.00676/) (2015)
- [Choquet regularization for reinforcement learning](https://mlsanity.com/paper/2208.08497/) (2022)

## Abstract (arXiv)

We consider the problem of risk-sensitive control in a reinforcement learning (RL) framework. In particular, we aim to find a risk-optimal policy by maximizing the distortion riskmetric (DRM) of the discounted reward in a finite horizon Markov decision process (MDP). DRMs are a rich class of risk measures that include several well-known risk measures as special cases. We derive a policy Hessian theorem for the DRM objective using the likelihood ratio method. Using this result, we propose a natural DRM Hessian estimator from sample trajectories of the underlying MDP. Next, we present a cubic-regularized policy Newton algorithm for solving this problem in an on-policy RL setting using estimates of the DRM gradient and Hessian. Our proposed algorithm is shown to converge to an ε-second-order stationary point (ε-SOSP) of the DRM objective, and this guarantee ensures the escaping of saddle points. The sample complexity of our algorithms to find an  ε-SOSP is O(ε-3.5). Our experiments validate the theoretical findings. To the best of our knowledge, our is the first work to present convergence to an ε-SOSP of a risk-sensitive objective, while existing works in the literature have either shown convergence to a first-order stationary point of a risk-sensitive objective, or a SOSP of a risk-neutral one.