# Robust Reinforcement Learning in POMDPs with Incomplete and Noisy Observations

arXiv: https://arxiv.org/abs/1902.05795 · HTML: https://mlsanity.com/paper/1902.05795/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Feb 2019.

## Built on by (most cited first, 10 of 10)

- [Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPs](https://mlsanity.com/paper/2110.05038/) (2021)
- [WCM: A World Critic Model for Vision-Language-Action Reinforcement Learning](https://mlsanity.com/paper/2607.29613/) (2026)
- [Deep Belief Markov Models for POMDP Inference](https://mlsanity.com/paper/2503.13438/) (2025)
- [Off-Policy Evaluation Under Nonignorable Missing Data](https://mlsanity.com/paper/2507.06961/) (2025)
- [A Scavenger Hunt for Service Robots](https://mlsanity.com/paper/2103.05225/) (2021)
- [Quantifying First-Order Markov Violations in Noisy Reinforcement Learning: A Causal Discovery Approach](https://mlsanity.com/paper/2503.00206/) (2025)
- [Missing Data Multiple Imputation for Tabular Q-Learning in Online RL](https://mlsanity.com/paper/2510.10709/) (2025)
- [Prediction-Based Markov Violation Scores for Detecting Non-Markovian Observations in Reinforcement Learning](https://mlsanity.com/paper/2603.27389/) (2026)
- [Missingness-MDPs: Bridging the Theory of Missing Data and POMDPs](https://mlsanity.com/paper/2605.12262/) (2026)
- [Admissable: Training Reinforcement Learning Agents against Adversarial Missingness](https://mlsanity.com/paper/2609.15297/) (2026)

## Abstract (arXiv)

In real-world scenarios, the observation data for reinforcement learning with continuous control is commonly noisy and part of it may be dynamically missing over time, which violates the assumption of many current methods developed for this. We addressed the issue within the framework of partially observable Markov Decision Process (POMDP) using a model-based method, in which the transition model is estimated from the incomplete and noisy observations using a newly proposed surrogate loss function with local approximation, while the policy and value function is learned with the help of belief imputation. For the latter purpose, a generative model is constructed and is seamlessly incorporated into the belief updating procedure of POMDP, which enables robust execution even under a significant incompleteness and noise. The effectiveness of the proposed method is verified on a collection of benchmark tasks, showing that our approach outperforms several compared methods under various challenging scenarios.