# On the Convergence of Modified Policy Iteration in Risk Sensitive Exponential Cost Markov Decision Processes

arXiv: https://arxiv.org/abs/2302.03811 · HTML: https://mlsanity.com/paper/2302.03811/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 26 references that are in the map.
- First posted on arXiv 8 Feb 2023.

## Builds on (most cited first, 7 of 7)

- [Risk-Sensitive Reinforcement Learning: Near-Optimal Risk-Sample Tradeoff in Regret](https://mlsanity.com/paper/2006.13827/) (2020)
- [Robust Markov Decision Process: Beyond Rectangularity](https://mlsanity.com/paper/1811.00215/) (2018)
- [Beyond the One Step Greedy Approach in Reinforcement Learning](https://mlsanity.com/paper/1802.03654/) (2018)
- [First-Order Methods for Wasserstein Distributionally Robust MDP](https://mlsanity.com/paper/2009.06790/) (2020)
- [A Policy Gradient Algorithm for the Risk-Sensitive Exponential Cost MDP](https://mlsanity.com/paper/2202.04157/) (2022)
- [Distributionally Robust Optimization for Sequential Decision Making](https://mlsanity.com/paper/1801.04745/) (2018)
- [RASR: Risk-Averse Soft-Robust MDPs with EVaR and Entropic Risk](https://mlsanity.com/paper/2209.04067/) (2022)

## Abstract (arXiv)

Modified policy iteration (MPI) is a dynamic programming algorithm that combines elements of policy iteration and value iteration. The convergence of MPI has been well studied in the context of discounted and average-cost MDPs. In this work, we consider the exponential cost risk-sensitive MDP formulation, which is known to provide some robustness to model parameters. Although policy iteration and value iteration have been well studied in the context of risk sensitive MDPs, MPI is unexplored. We provide the first proof that MPI also converges for the risk-sensitive problem in the case of finite state and action spaces. Since the exponential cost formulation deals with the multiplicative Bellman equation, our main contribution is a convergence proof which is quite different than existing results for discounted and risk-neutral average-cost problems as well as risk sensitive value and policy iteration approaches. We conclude our analysis with simulation results, assessing MPI's performance relative to alternative dynamic programming methods like value iteration and policy iteration across diverse problem parameters. Our findings highlight risk-sensitive MPI's enhanced computational efficiency compared to both value and policy iteration techniques.