# A Policy Gradient Algorithm for the Risk-Sensitive Exponential Cost MDP

arXiv: https://arxiv.org/abs/2202.04157 · HTML: https://mlsanity.com/paper/2202.04157/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Feb 2022.

## Built on by (most cited first, 13 of 13)

- [Pessimism Meets Risk: Risk-Sensitive Offline Reinforcement Learning](https://mlsanity.com/paper/2407.07631/) (2024)
- [Recursive Entropic Risk Optimization in Discounted MDPs: Sample Complexity Bounds with a Generative Model](https://mlsanity.com/paper/2506.00286/) (2025)
- [Markov Chain Variance Estimation: A Stochastic Approximation Approach](https://mlsanity.com/paper/2409.05733/) (2024)
- [Planning and Learning in Average Risk-aware MDPs](https://mlsanity.com/paper/2503.17629/) (2025)
- [Conformal Prediction Beyond the Horizon: Distribution-Free Inference for Policy Evaluation](https://mlsanity.com/paper/2510.26026/) (2025)
- [Risk-Sensitive Multi-Agent Reinforcement Learning in Network Aggregative Markov Games](https://mlsanity.com/paper/2402.05906/) (2024)
- [An Actor-Critic Algorithm with Function Approximation for Risk Sensitive Cost Markov Decision Processes](https://mlsanity.com/paper/2502.11604/) (2025)
- [Reinforcement Learning for Exponential Utility: Algorithms and Convergence in Discounted MDPs](https://mlsanity.com/paper/2605.08053/) (2026)
- [On the Sample Complexity of Discounted Reinforcement Learning with Optimized Certainty Equivalents](https://mlsanity.com/paper/2605.21763/) (2026)
- [On the Convergence of Modified Policy Iteration in Risk Sensitive Exponential Cost Markov Decision Processes](https://mlsanity.com/paper/2302.03811/) (2023)
- [Risk-Averse Reinforcement Learning with Itakura-Saito Loss](https://mlsanity.com/paper/2505.16925/) (2025)
- [Long-Run Conditional Value-at-Risk Reinforcement Learning](https://mlsanity.com/paper/2603.09734/) (2026)
- [Near-Optimal Sample Complexity for Recursive Entropic Risk Reinforcement Learning with a Generative Model](https://mlsanity.com/paper/2610.06931/) (2026)

## Abstract (arXiv)

We study the risk-sensitive exponential cost MDP formulation and develop a trajectory-based gradient algorithm to find the stationary point of the cost associated with a set of parameterized policies. We derive a formula that can be used to compute the policy gradient from (state, action, cost) information collected from sample paths of the MDP for each fixed parameterized policy. Unlike the traditional average-cost problem, standard stochastic approximation theory cannot be used to exploit this formula. To address the issue, we introduce a truncated and smooth version of the risk-sensitive cost and show that this new cost criterion can be used to approximate the risk-sensitive cost and its gradient uniformly under some mild assumptions. We then develop a trajectory-based gradient algorithm to minimize the smooth truncated estimation of the risk-sensitive cost and derive conditions under which a sequence of truncations can be used to solve the original, untruncated cost problem.