# Markov Decision Processes under Model Uncertainty

arXiv: https://arxiv.org/abs/2206.06109 · HTML: https://mlsanity.com/paper/2206.06109/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 83 references that are in the map.
- First posted on arXiv 13 Jun 2022.

## Builds on (most cited first, 24 of 24)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Recent Advances in Reinforcement Learning in Finance](https://mlsanity.com/paper/2112.04553/) (2021)
- [Deep Reinforcement Learning for Trading](https://mlsanity.com/paper/1911.10107/) (2019)
- [Practical Deep Reinforcement Learning Approach for Stock Trading](https://mlsanity.com/paper/1811.07522/) (2018)
- [Deep Hedging of Derivatives Using Reinforcement Learning](https://mlsanity.com/paper/2103.16409/) (2021)
- [Distributionally Robust Mean-Variance Portfolio Selection with Wasserstein Distances](https://mlsanity.com/paper/1802.04885/) (2018)
- [On Linear Optimization over Wasserstein Balls](https://mlsanity.com/paper/2004.07162/) (2020)
- [QLBS: Q-Learner in the Black-Scholes(-Merton) Worlds](https://mlsanity.com/paper/1712.04609/) (2017)
- [Model-based Deep Reinforcement Learning for Dynamic Portfolio Optimization](https://mlsanity.com/paper/1901.08740/) (2019)
- [Multi-Period Trading via Convex Optimization](https://mlsanity.com/paper/1705.00109/) (2017)
- [Reinforcement Learning for Mean Field Games, with Applications to Economics](https://mlsanity.com/paper/2106.13755/) (2021)
- [Distributionally Robust Optimization for Sequential Decision Making](https://mlsanity.com/paper/1801.04745/) (2018)
- [Reinforcement Learning for Portfolio Management](https://mlsanity.com/paper/1909.09571/) (2019)
- [Reinforcement Learning Algorithm for Mixed Mean Field Control Games](https://mlsanity.com/paper/2205.02330/) (2022)
- [Robust Markowitz mean-variance portfolio selection under ambiguous covariance matrix *](https://mlsanity.com/paper/1610.06805/) (2016)
- [Detecting data-driven robust statistical arbitrage strategies with deep neural networks](https://mlsanity.com/paper/2203.03179/) (2022)
- [Robust Optimal Control Using Conditional RiskMappings in Infinite Horizon](https://mlsanity.com/paper/1806.00983/) (2018)
- [Duality Theory for Robust Utility Maximisation](https://mlsanity.com/paper/2007.08376/) (2020)
- [Distributionally robust portfolio maximisation and marginal utility pricing in one period financial markets](https://mlsanity.com/paper/2105.00935/) (2021)
- [Exponential utility maximization under model uncertainty for unbounded endowments](https://mlsanity.com/paper/1610.00999/) (2016)
- [Nonconcave Robust Optimization with Discrete Strategies under Knightian Uncertainty](https://mlsanity.com/paper/1711.03875/) (2017)
- [Portfolio diversification and model uncertainty: a robust dynamic mean-variance approach](https://mlsanity.com/paper/1809.01464/) (2018)
- [Robust consumption-investment problem Under CRRA and CARA utilities with time-varying confidence sets](https://mlsanity.com/paper/1811.12149/) (2018)
- [Robust utility maximization under model uncertainty via a penalization approach](https://mlsanity.com/paper/1907.13345/) (2019)

## Built on by (most cited first, 3 of 3)

- [Q-Learning under Finite Model Uncertainty](https://mlsanity.com/paper/2407.04259/) (2024)
- [Distributionally Robust Deep Q-Learning](https://mlsanity.com/paper/2505.19058/) (2025)
- [Policy Gradient Learning for Distributionally Robust Markov Decision Processes under Wasserstein Ambiguity](https://mlsanity.com/paper/2606.27610/) (2026)

## Abstract (arXiv)

We introduce a general framework for Markov decision problems under model uncertainty in a discrete-time infinite horizon setting. By providing a dynamic programming principle we obtain a local-to-global paradigm, namely solving a local, i.e., a one time-step robust optimization problem leads to an optimizer of the global (i.e. infinite time-steps) robust stochastic optimal control problem, as well as to a corresponding worst-case measure. Moreover, we apply this framework to portfolio optimization involving data of the S&P 500. We present two different types of ambiguity sets; one is fully data-driven given by a Wasserstein-ball around the empirical measure, the second one is described by a parametric set of multivariate normal distributions, where the corresponding uncertainty sets of the parameters are estimated from the data. It turns out that in scenarios where the market is volatile or bearish, the optimal portfolio strategies from the corresponding robust optimization problem outperforms the ones without model uncertainty, showcasing the importance of taking model uncertainty into account.