# Data-driven robust Markov decision processes on Borel spaces: performance guarantees via an axiomatic approach

arXiv: https://arxiv.org/abs/2603.08979 · HTML: https://mlsanity.com/paper/2603.08979/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 67 references that are in the map.
- First posted on arXiv 9 Mar 2026.

## Builds on (most cited first, 21 of 21)

- [Sample Complexity of Robust Reinforcement Learning with a Generative Model](https://mlsanity.com/paper/2112.01506/) (2021)
- [The Curious Price of Distributional Robustness in Reinforcement Learning with a Generative Model](https://mlsanity.com/paper/2305.16589/) (2023)
- [Improved Sample Complexity Bounds for Distributionally Robust Reinforcement Learning](https://mlsanity.com/paper/2303.02783/) (2023)
- [From Data to Decisions: Distributionally Robust Optimization is Optimal](https://mlsanity.com/paper/1704.04118/) (2017)
- [Wasserstein Distributionally Robust Stochastic Control: A Data-Driven Approach](https://mlsanity.com/paper/1812.09808/) (2018)
- [On Linear Optimization over Wasserstein Balls](https://mlsanity.com/paper/2004.07162/) (2020)
- [Towards Minimax Optimality of Model-based Robust Reinforcement Learning](https://mlsanity.com/paper/2302.05372/) (2023)
- [Overcoming the Sim-to-Real Gap: Leveraging Simulation to Learn to Explore for Real-World RL](https://mlsanity.com/paper/2410.20254/) (2024)
- [On the Foundation of Distributionally Robust Reinforcement Learning](https://mlsanity.com/paper/2311.09018/) (2023)
- [Distributionally Robust Model-based Reinforcement Learning with Large State Spaces](https://mlsanity.com/paper/2309.02236/) (2023)
- [Distributionally Robust Markov Decision Processes and their Connection to Risk Measures](https://mlsanity.com/paper/2007.13103/) (2020)
- [Certified Robust Neural Networks: Generalization and Corruption Resistance](https://mlsanity.com/paper/2303.02251/) (2023)
- [Data-driven ambiguity sets with probabilistic guarantees for dynamic processes](https://mlsanity.com/paper/1909.11194/) (2019)
- [Wasserstein Distributionally Robust Motion Control for Collision Avoidance Using Conditional Value-at-Risk](https://mlsanity.com/paper/2001.04727/) (2020)
- [Distributional robustness in minimax linear quadratic control with Wasserstein distance](https://mlsanity.com/paper/2102.12715/) (2021)
- [Robustness to incorrect system models in stochastic control](https://mlsanity.com/paper/1803.06046/) (2018)
- [Wasserstein Distributionally Robust Control of Partially Observable Linear Stochastic Systems](https://mlsanity.com/paper/2212.04644/) (2022)
- [Distributionally Robust Differential Dynamic Programming with Wasserstein Distance](https://mlsanity.com/paper/2305.09760/) (2023)
- [Bounding the Difference between the Values of Robust and Non-Robust Markov Decision Problems](https://mlsanity.com/paper/2308.05520/) (2023)
- [Infinite Horizon Average Cost Dynamic Programming Subject to Total Variation Distance Ambiguity](https://mlsanity.com/paper/1512.06510/) (2015)
- [Model approximation in MDPs with unbounded per-step cost](https://mlsanity.com/paper/2402.08813/) (2024)

## Abstract (arXiv)

We consider Markov decision processes (MDPs) with unknown disturbance distribution and address this problem using the robust Markov decision process (RMDP) approach. We construct the empirical distribution of the unknown disturbance distribution and characterize our ambiguity set of distributions as the sublevel set of a nonnegative distance function from the empirical distribution. By connecting the weak convergence of distributions to convergence with respect to the distance function, we prove that the robust optimal value function and the out-of-sample value function converge to the true optimal value function with increasing sample-sizes. We establish that, for finite sample-sizes, the robust optimal value function serves as a high probability upper bound on the out-of-sample value function. We also obtain probabilistic convergence rates, sample complexity bounds, and out-of-distribution performance bounds. The finite sample performance guarantees rely on the distance function satisfying a certain concentration type inequality. Several well-studied distances in the literature meet the requirements imposed on the distance function. We also analyze the data-driven properties of empirical MDPs and demonstrate that, unlike our data-driven RMDPs, empirical MDPs fail to satisfy some of the finite sample performance guarantees.