Data to 7 Oct 2026

Off-Policy Evaluation in Partially Observable Environments

arXiv 1909.03739 ↗First posted 9 Sep 2019cs.LG cs.AI cs.SYcited by 65

Not on the map. Why?

Abstract (arXiv)

This work studies the problem of batch off-policy evaluation for Reinforcement Learning in partially observable environments. Off-policy evaluation under partial observability is inherently prone to bias, with risk of arbitrarily large errors. We define the problem of off-policy evaluation for Partially Observable Markov Decision Processes (POMDPs) and establish what we believe is the first off-policy evaluation result for POMDPs. In addition, we formulate a model in which observed and unobserved variables are decoupled into two dynamic processes, called a Decoupled POMDP. We show how off-policy evaluation can be performed under this new model, mitigating estimation errors inherent to general POMDPs. We demonstrate the pitfalls of off-policy evaluation in POMDPs using a well-known off-policy method, Importance Sampling, and compare it with our result on synthetic medical data.

Key facts. Cited by 65 papers in the map (data through 7 Oct 2026). First posted on arXiv 9 Sep 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 1; 2020: 0; 2021: 16; 2022: 17; 2023: 10; 2024: 8; 2025: 8; 2026: 5.

65 papers.

Sort:

By area:

2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 92
2019Task-Relevant Adversarial Imitation Learningcited by 45
2021Causal Inference Under Unmeasured Confounding With Negative Controls: A Minimax Learning Approachcited by 43
2021Universal Off-Policy Evaluationcited by 43
2023A Survey on Causal Reinforcement Learningcited by 43
2021Proximal Reinforcement Learning: Efficient Off-Policy Evaluation in Partially Observed Markov Decision Processescited by 40
2021Deep Proxy Causal Learning and its Application to Confounded Bandit Policy Evaluationcited by 39
2021Causal Reinforcement Learning using Observational and Interventional Datacited by 38
2021Instrumental Variable Value Iteration for Causal Offline Reinforcement Learningcited by 35
2022Provably Efficient Reinforcement Learning in Partially Observable Dynamical Systemscited by 35
2021A Minimax Learning Approach to Off-Policy Evaluation in Confounded Partially Observable Markov Decision Processescited by 34
2022Off-Policy Confidence Interval Estimation with Confounded Markov Decision Processcited by 31

38 more

2022Long-term Causal Inference Under Persistent Confounding via Data Combinationcited by 27
2022Pessimism in the Face of Confounders: Provably Efficient Offline Reinforcement Learning in Partially Observable Markov Decision Processescited by 26
2021Off-Policy Evaluation in Partially Observed Markov Decision Processes under Sequential Ignorabilitycited by 24
2021A Spectral Approach to Off-Policy Evaluation for POMDPscited by 20
2022Future-Dependent Value-Based Off-Policy Evaluation in POMDPscited by 20
2022An Instrumental Variable Approach to Confounded Off-Policy Evaluationcited by 20
2022Off-Policy Evaluation for Episodic Partially Observable Markov Decision Processes under Non-Parametric Modelscited by 18
2021Invariant Policy Learning: A Causal Perspectivecited by 17
2021On Covariate Shift of Latent Confounders in Imitation and Reinforcement Learningcited by 14
2022Model-Free and Model-Based Policy Evaluation when Causality is Uncertaincited by 12
2022Offline Policy Evaluation and Optimization under Confoundingcited by 12
2023Robust Fitted-Q-Evaluation and Iteration under Sequentially Exogenous Unobserved Confounderscited by 12
2023Finding Counterfactually Optimal Action Sequences in Continuous State Spacescited by 12
2024A CMDP-within-online framework for Meta-Safe Reinforcement Learningcited by 12
2021Direct Advantage Estimationcited by 11
2023Reinforcement Learning with History-Dependent Dynamic Contextscited by 11
2023Comparing Causal Frameworks: Potential Outcomes, Structural Models, Graphs, and Abstractionscited by 11
2021Learning-to-defer for sequential medical decision-making under uncertaintycited by 10
2022A Survey of Deep Causal Models and Their Industrial Applicationscited by 10
2023Safe Policy Improvement for POMDPs via Finite-State Controllerscited by 10
2025Offline Reinforcement Learning in Large State Spaces: Algorithms and Guaranteescited by 10
2021Training a Resilient Q-Network against Observational Interferencecited by 9
2022Strategic Decision-Making in the Presence of Information Asymmetry: Provably Efficient RL with Algorithmic Instrumentscited by 9
2024On the Curses of Future and History in Future-dependent Value Functions for Off-policy Evaluationcited by 9
2023Source Condition Double Robust Inference on Functionals of Inverse Problemscited by 8
2025Statistical Tractability of Off-policy Evaluation of History-dependent Policies in POMDPscited by 7
2021Ambiguous Dynamic Treatment Regimes: A Reinforcement Learning Approachcited by 6
2025Automatic Reward Shaping from Confounded Offline Datacited by 6
2022A Reinforcement Learning Approach to Estimating Long-term Treatment Effectscited by 5
2023HOPE: Human-Centric Off-Policy Evaluation for E-Learning and Healthcarecited by 5
2024RL in Latent MDPs is Tractable: Online Guarantees via Off-Policy Evaluationcited by 5
2024Two-way Deconfounder for Off-policy Evaluation in Causal Reinforcement Learningcited by 5
2022Generalizing Off-Policy Evaluation From a Causal Perspective For Sequential Decision-Makingcited by 4
2021Offline Policy Comparison under Limited Historical Agent-Environment Interactionscited by 3
2022Blessing from Human-AI Interaction: Super Reinforcement Learning in Confounded Environmentscited by 3
2022Causal Deep Reinforcement Learning Using Observational Datacited by 3
2024OPERA: Automatic Offline Policy Evaluation with Re-weighted Aggregates of Multiple Estimatorscited by 3
2026CIVeX: Causal Intervention Verification for Language Agentscited by 3

50 most cited of 65 All 65 (JSON)