# Model approximation in MDPs with unbounded per-step cost

arXiv: https://arxiv.org/abs/2402.08813 · HTML: https://mlsanity.com/paper/2402.08813/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 45 references that are in the map.
- First posted on arXiv 13 Feb 2024.

## Builds on (most cited first, 10 of 10)

- [DeepMDP: Learning Continuous Latent Space Models for Representation Learning](https://mlsanity.com/paper/1906.02736/) (2019)
- [Approximate information state for approximate planning and reinforcement learning in partially observed systems](https://mlsanity.com/paper/2010.08843/) (2020)
- [Near Optimal Behavior via Approximate State Abstraction](https://mlsanity.com/paper/1701.04113/) (2017)
- [Plannable Approximations to MDP Homomorphisms: Equivariance under Actions](https://mlsanity.com/paper/2002.11963/) (2020)
- [Near Optimality of Finite Memory Feedback Policies in Partially Observed Markov Decision Processes](https://mlsanity.com/paper/2010.07452/) (2020)
- [On overfitting and asymptotic bias in batch reinforcement learning with partial observability](https://mlsanity.com/paper/1709.07796/) (2017)
- [Asymptotic Optimality of Finite Approximations to Markov Decision Processes with Borel Spaces](https://mlsanity.com/paper/1503.02244/) (2015)
- [Robustness to incorrect system models in stochastic control](https://mlsanity.com/paper/1803.06046/) (2018)
- [Near Optimality of Quantized Policies in Stochastic Control Under Weak Continuity Conditions](https://mlsanity.com/paper/1410.6985/) (2014)
- [Robustness to incorrect priors in partially observed stochastic control](https://mlsanity.com/paper/1803.05103/) (2018)

## Built on by (most cited first, 1 of 1)

- [Data-driven robust Markov decision processes on Borel spaces: performance guarantees via an axiomatic approach](https://mlsanity.com/paper/2603.08979/) (2026)

## Abstract (arXiv)

We consider the problem of designing a control policy for an infinite-horizon discounted cost Markov decision process M when we only have access to an approximate model hatM. How well does an optimal policy hatπstar of the approximate model perform when used in the original model M? We answer this question by bounding a weighted norm of the difference between the value function of hatπ^star  when used in M and the optimal value function of M. We then extend our results and obtain potentially tighter upper bounds by considering affine transformations of the per-step cost. We further provide upper bounds that explicitly depend on the weighted distance between cost functions and weighted distance between transition kernels of the original and approximate models. We present examples to illustrate our results.