# Robust Phi-Divergence MDPs

arXiv: https://arxiv.org/abs/2205.14202 · HTML: https://mlsanity.com/paper/2205.14202/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 72 references that are in the map.
- First posted on arXiv 27 May 2022.

## Builds on (most cited first, 21 of 21)

- [CVXPY: A Python-Embedded Modeling Language for Convex Optimization](https://mlsanity.com/paper/1603.00943/) (2016)
- [Online Robust Reinforcement Learning with Model Uncertainty](https://mlsanity.com/paper/2109.14523/) (2021)
- [Sample Complexity of Robust Reinforcement Learning with a Generative Model](https://mlsanity.com/paper/2112.01506/) (2021)
- [Policy Gradient Method For Robust Reinforcement Learning](https://mlsanity.com/paper/2205.07344/) (2022)
- [Reinforcement Learning under Model Mismatch](https://mlsanity.com/paper/1706.04711/) (2017)
- [Partial Policy Iteration for L1-Robust Markov Decision Processes](https://mlsanity.com/paper/2006.09484/) (2020)
- [Twice regularized MDPs and the equivalence between robustness and regularization](https://mlsanity.com/paper/2110.06267/) (2021)
- [From Data to Decisions: Distributionally Robust Optimization is Optimal](https://mlsanity.com/paper/1704.04118/) (2017)
- [Distributionally Robust Counterpart in Markov Decision Processes](https://mlsanity.com/paper/1501.07418/) (2015)
- [Navigating to the Best Policy in Markov Decision Processes](https://mlsanity.com/paper/2106.02847/) (2021)
- [Scalable First-Order Methods for Robust MDPs](https://mlsanity.com/paper/2005.05434/) (2020)
- [Robust Markov Decision Process: Beyond Rectangularity](https://mlsanity.com/paper/1811.00215/) (2018)
- [First-Order Methods for Wasserstein Distributionally Robust MDP](https://mlsanity.com/paper/2009.06790/) (2020)
- [Projection onto the capped simplex](https://mlsanity.com/paper/1503.01002/) (2015)
- [Distributionally Robust Optimization for Sequential Decision Making](https://mlsanity.com/paper/1801.04745/) (2018)
- [Robust Policies For Proactive ICU Transfers](https://mlsanity.com/paper/2002.06247/) (2020)
- [Projections onto the canonical simplex with additional linear inequalities](https://mlsanity.com/paper/1905.03488/) (2019)
- [Fast Approximate Dynamic Programming for Infinite-Horizon Markov Decision Processes](https://mlsanity.com/paper/2102.08880/) (2021)
- [Fast Projection onto the Capped Simplex with Applications to Sparse Regression in Bioinformatics](https://mlsanity.com/paper/2110.08471/) (2021)
- [Play to Grade: Testing Coding Games as Classifying Markov Decision Process](https://mlsanity.com/paper/2110.14615/) (2021)
- [Distributionally robust inventory control when demand is a martingale](https://mlsanity.com/paper/1511.09437/) (2015)

## Built on by (most cited first, 1 of 1)

- [Twice Regularized Markov Decision Processes: The Equivalence between Robustness and Regularization](https://mlsanity.com/paper/2303.06654/) (2023)

## Abstract (arXiv)

In recent years, robust Markov decision processes (MDPs) have emerged as a prominent modeling framework for dynamic decision problems affected by uncertainty. In contrast to classical MDPs, which only account for stochasticity by modeling the dynamics through a stochastic process with a known transition kernel, robust MDPs additionally account for ambiguity by optimizing in view of the most adverse transition kernel from a prescribed ambiguity set. In this paper, we develop a novel solution framework for robust MDPs with s-rectangular ambiguity sets that decomposes the problem into a sequence of robust Bellman updates and simplex projections. Exploiting the rich structure present in the simplex projections corresponding to phi-divergence ambiguity sets, we show that the associated s-rectangular robust MDPs can be solved substantially faster than with state-of-the-art commercial solvers as well as a recent first-order solution scheme, thus rendering them attractive alternatives to classical MDPs in practical applications.