# Policy Gradient Learning for Distributionally Robust Markov Decision Processes under Wasserstein Ambiguity

arXiv: https://arxiv.org/abs/2606.27610 · HTML: https://mlsanity.com/paper/2606.27610/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 18 of its 49 references that are in the map.
- First posted on arXiv 25 Jun 2026.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 18 of 18)

- [Data-driven Distributionally Robust Optimization Using the Wasserstein Metric: Performance Guarantees and Tractable Reformulations](https://mlsanity.com/paper/1505.05116/) (2015)
- [Distributionally Robust Stochastic Optimization with Wasserstein Distance](https://mlsanity.com/paper/1604.02199/) (2016)
- [Policy Gradient Method For Robust Reinforcement Learning](https://mlsanity.com/paper/2205.07344/) (2022)
- [Policy Gradient in Robust MDPs with Global Convergence Guarantee](https://mlsanity.com/paper/2212.10439/) (2022)
- [Policy Gradient for Rectangular Robust Markov Decision Processes](https://mlsanity.com/paper/2301.13589/) (2023)
- [Wasserstein Distributionally Robust Stochastic Control: A Data-Driven Approach](https://mlsanity.com/paper/1812.09808/) (2018)
- [Distributionally Robust Mean-Variance Portfolio Selection with Wasserstein Distances](https://mlsanity.com/paper/1802.04885/) (2018)
- [On Linear Optimization over Wasserstein Balls](https://mlsanity.com/paper/2004.07162/) (2020)
- [Robust Risk-Aware Reinforcement Learning](https://mlsanity.com/paper/2108.10403/) (2021)
- [First-Order Methods for Wasserstein Distributionally Robust MDP](https://mlsanity.com/paper/2009.06790/) (2020)
- [Sensitivity analysis of Wasserstein distributionally robust optimization problems](https://mlsanity.com/paper/2006.12022/) (2020)
- [Wasserstein Distributionally Robust Policy Evaluation and Learning for Contextual Bandits](https://mlsanity.com/paper/2309.08748/) (2023)
- [Distributional robustness in minimax linear quadratic control with Wasserstein distance](https://mlsanity.com/paper/2102.12715/) (2021)
- [Markov Decision Processes under Model Uncertainty](https://mlsanity.com/paper/2206.06109/) (2022)
- [Robust Reinforcement Learning with Dynamic Distortion Risk Measures](https://mlsanity.com/paper/2409.10096/) (2024)
- [Deep Neural Operator Learning for Probabilistic Models](https://mlsanity.com/paper/2511.07235/) (2025)
- [First order Martingale model risk and semi-static hedging](https://mlsanity.com/paper/2410.06906/) (2024)
- [Sensitivity Analysis of Distributionally Robust BSDEs and RBSDEs](https://mlsanity.com/paper/2511.01828/) (2025)

## Abstract (arXiv)

We study finite-horizon Markov decision processes under distributional uncertainty in the transition kernels and develop a policy-gradient framework for Wasserstein distributionally robust control. Ambiguity is modeled by Wasserstein balls of common radius centered at state--action-dependent nominal transition kernels, leading to a max--min problem over randomized policies and admissible transition laws. Because the worst-case transition law depends implicitly on the policy parameters, the standard policy-gradient argument does not apply directly. We address this difficulty by combining the dynamic programming recursion with Wasserstein duality and a primal envelope argument. In general, the right and left directional derivatives of the one-step worst-case value are obtained by taking the minimum or maximum expected downstream value derivative over the set of worst-case transition laws. In finite state--action spaces, this set is characterized through the optimal face of a transport linear program, yielding an exact directional-derivative recursion. Under the required stability conditions and uniqueness of the active dual and transport optimizers, the derivative becomes linear in the policy perturbation and admits an explicit vector valued policy-gradient recursion. Building on this representation, we propose a robust actor--critic implementation and evaluate it on benchmark examples.