# Towards Efficient Risk-Sensitive Policy Gradient: An Iteration Complexity Analysis

arXiv: https://arxiv.org/abs/2403.08955 · HTML: https://mlsanity.com/paper/2403.08955/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 26 of its 47 references that are in the map.
- First posted on arXiv 13 Mar 2024.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 26 of 26)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [Safe Model-based Reinforcement Learning with Stability Guarantees](https://mlsanity.com/paper/1705.08551/) (2017)
- [Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Tasks](https://mlsanity.com/paper/2306.13831/) (2023)
- [Better Theory for SGD in the Nonconvex World](https://mlsanity.com/paper/2002.03329/) (2020)
- [Non-convex Optimization for Machine Learning](https://mlsanity.com/paper/1712.07897/) (2017)
- [Risk-sensitive Reinforcement Learning](https://mlsanity.com/paper/1311.2097/) (2013)
- [Sample Complexity of Episodic Fixed-Horizon Reinforcement Learning](https://mlsanity.com/paper/1510.08906/) (2015)
- [Stochastic Variance-Reduced Policy Gradient](https://mlsanity.com/paper/1806.05618/) (2018)
- [Unified Optimal Analysis of the (Stochastic) Gradient Method](https://mlsanity.com/paper/1907.04232/) (2019)
- [A general sample complexity analysis of vanilla policy gradient](https://mlsanity.com/paper/2107.11433/) (2021)
- [Sample Efficient Policy Gradient Methods with Recursive Variance Reduction](https://mlsanity.com/paper/1909.08610/) (2019)
- [An Improved Convergence Analysis of Stochastic Variance-Reduced Policy Gradient](https://mlsanity.com/paper/1905.12615/) (2019)
- [Reinforcement Learning in Economics and Finance](https://mlsanity.com/paper/2003.10014/) (2020)
- [Risk-Sensitive Reinforcement Learning: Near-Optimal Risk-Sample Tradeoff in Regret](https://mlsanity.com/paper/2006.13827/) (2020)
- [Learning Bounds for Risk-sensitive Learning](https://mlsanity.com/paper/2006.08138/) (2020)
- [RMIX: Learning Risk-Sensitive Policies for Cooperative Reinforcement Learning Agents](https://mlsanity.com/paper/2102.08159/) (2021)
- [The Sample-Complexity of General Reinforcement Learning](https://mlsanity.com/paper/1308.4828/) (2013)
- [Epistemic Risk-Sensitive Reinforcement Learning](https://mlsanity.com/paper/1906.06273/) (2019)
- [Risk-Sensitive Reinforcement Learning with Exponential Criteria](https://mlsanity.com/paper/2212.09010/) (2022)
- [Reinforcement Learning for Portfolio Management](https://mlsanity.com/paper/1909.09571/) (2019)
- [Reinforcement Learning Algorithms: An Overview and Classification](https://mlsanity.com/paper/2209.14940/) (2022)
- [Risk-Aware Submodular Optimization for Multi-Robot Coordination](https://mlsanity.com/paper/2003.10492/) (2020)
- [Data-Driven Distributionally Robust Optimal Control with State-Dependent Noise](https://mlsanity.com/paper/2303.02293/) (2023)

## Built on by (most cited first, 6 of 6)

- [CAML: Collaborative Auxiliary Modality Learning for Multi-Agent Systems](https://mlsanity.com/paper/2502.17821/) (2025)
- [Adaptive Conformal Guidance for Learning under Uncertainty](https://mlsanity.com/paper/2502.16736/) (2025)
- [MMCD: Multi-Modal Collaborative Decision-Making for Connected Autonomy with Knowledge Distillation](https://mlsanity.com/paper/2509.18198/) (2025)
- [Quantum-Inspired Reinforcement Learning in the Presence of Epistemic Ambivalence](https://mlsanity.com/paper/2503.04219/) (2025)
- [Active Asymmetric Multi-Agent Multimodal Learning under Uncertainty](https://mlsanity.com/paper/2602.04763/) (2026)
- [CARE: Certifying Acceleration for Vision-Language-Action Inference](https://mlsanity.com/paper/2610.08917/) (2026)

## Abstract (arXiv)

Reinforcement Learning (RL) has shown exceptional performance across various applications, enabling autonomous agents to learn optimal policies through interaction with their environments. However, traditional RL frameworks often face challenges in terms of iteration efficiency and safety. Risk-sensitive policy gradient methods, which incorporate both expected return and risk measures, have been explored for their ability to yield safe policies, yet their iteration complexity remains largely underexplored. In this work, we conduct a rigorous iteration complexity analysis for the risk-sensitive policy gradient method, focusing on the REINFORCE algorithm with an exponential utility function. We establish an iteration complexity of O(ε-2) to reach an ε-approximate first-order stationary point (FOSP). Furthermore, we investigate whether risk-sensitive algorithms can achieve better iteration complexity compared to their risk-neutral counterparts. Our analysis indicates that risk-sensitive REINFORCE can potentially converge faster. To validate our analysis, we empirically evaluate the learning performance and convergence efficiency of the risk-neutral and risk-sensitive REINFORCE algorithms in multiple environments: CartPole, MiniGrid, and Robot Navigation. Empirical results confirm that risk-sensitive cases can converge and stabilize faster compared to their risk-neutral counterparts. More details can be found on our website https://anonymous.4open.science/w/riskrl.