# Partially Observable Total-Cost Markov Decision Processes with Weakly Continuous Transition Probabilities

arXiv: https://arxiv.org/abs/1401.2168 · HTML: https://mlsanity.com/paper/1401.2168/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Jan 2014.

## Built on by (most cited first, 6 of 6)

- [Convergence of Finite Memory Q-Learning for POMDPs and Near Optimality of Learned Policies under Filter Stability](https://mlsanity.com/paper/2103.12158/) (2021)
- [Q-Learning for MDPs with General Spaces: Convergence and Near Optimality via Quantization under Weak Continuity](https://mlsanity.com/paper/2111.06781/) (2021)
- [Q-Learning for Stochastic Control under General Information Structures and Non-Markovian Environments](https://mlsanity.com/paper/2311.00123/) (2023)
- [Multi-Step Prediction in Linearized Latent State Spaces for Representation Learning](https://mlsanity.com/paper/2209.01127/) (2022)
- [Refined Bounds on Near Optimality Finite Window Policies in POMDPs and Their Reinforcement Learning](https://mlsanity.com/paper/2409.04351/) (2024)
- [Computing optimal policies for managing inventories with noisy observations](https://mlsanity.com/paper/2507.19765/) (2025)

## Abstract (arXiv)

This paper describes sufficient conditions for the existence of optimal policies for Partially Observable Markov Decision Processes (POMDPs) with Borel state, observation, and action sets and with the expected total costs. Action sets may not be compact and one-step cost functions may be unbounded. The introduced conditions are also sufficient for the validity of optimality equations, semi-continuity of value functions, and convergence of value iterations to optimal values. Since POMDPs can be reduced to Completely Observable Markov Decision Processes (COMDPs), whose states are posterior state distributions, this paper focuses on the validity of the above mentioned optimality properties for COMDPs. The central question is whether transition probabilities for a COMDP are weakly continuous. We introduce sufficient conditions for this and show that the transition probabilities for a COMDP are weakly continuous, if transition probabilities of the underlying Markov Decision Process are weakly continuous and observation probabilities for the POMDP are continuous in the total variation. Moreover, the continuity in the total variation of the observation probabilities cannot be weakened to setwise continuity. The results are illustrated with counterexamples and examples.