mlsanityData to 14 Oct 2026

Proximal Reinforcement Learning: Efficient Off-Policy Evaluation in Partially Observed Markov Decision Processes

arXiv 2110.15332 ↗First posted 28 Oct 2021cs.LG math.OC math.STcited by 41

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

In applications of offline reinforcement learning to observational data, such as in healthcare or education, a general concern is that observed actions might be affected by unobserved factors, inducing confounding and biasing estimates derived under the assumption of a perfect Markov decision process (MDP) model. Here we tackle this by considering off-policy evaluation in a partially observed MDP (POMDP). Specifically, we consider estimating the value of a given target policy in a POMDP given trajectories with only partial state observations generated by a different and unknown policy that may depend on the unobserved state. We tackle two questions: what conditions allow us to identify the target policy value from the observed data and, given identification, how to best estimate it. To answer these, we extend the framework of proximal causal inference to our POMDP setting, providing a variety of settings where identification is made possible by the existence of so-called bridge functions. We then show how to construct semiparametrically efficient estimators in these settings. We term the resulting framework proximal reinforcement learning (PRL). We demonstrate the benefits of PRL in an extensive simulation study and on the problem of sepsis management.

Key facts. Cited by 41 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 31 of its 48 references that are in the map. First posted on arXiv 28 Oct 2021.

Builds on

31 of 48 references.

Closest ancestors · 2

Same direction, newest first

  1. 2021Instrumental Variable Value Iteration for Causal Offline Reinforcement Learning19 Feb
  2. 2020Confounding-Robust Policy Evaluation in Infinite-Horizon Reinforcement Learning11 Feb

Other references · 29

  1. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  2. 2016Identifying Causal Effects With Proxy Variables of an Unmeasured Confoundercited by 139
  3. 2019Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processescited by 119
  4. 2019Counterfactual Off-Policy Evaluation with Gumbel-Max Structural Causal Modelscited by 117
  5. 2020An Introduction to Proximal Causal Learningcited by 95
  6. 2020Minimax Estimation of Conditional Moment Modelscited by 83
  7. 2019Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learningcited by 75
  8. 2019Off-Policy Evaluation in Partially Observable Environmentscited by 66
  9. 2020Off-policy Policy Evaluation For Sequential Decisions Under Unobserved Confoundingcited by 61
  10. 2020Semiparametric proximal causal inferencecited by 59
  11. 2016Reinforcement Learning of POMDPs using Spectral Methodscited by 52
  12. 2021Causal Inference Under Unmeasured Confounding With Negative Controls: A Minimax Learning Approachcited by 44
17 more
  1. 2021Universal Off-Policy Evaluationcited by 43
  2. 2020Off-policy Evaluation in Infinite-Horizon Reinforcement Learning with Latent Confounderscited by 41
  3. 2020Provably Efficient Causal Reinforcement Learning with Confounded Observational Datacited by 40
  4. 2021Deep Proxy Causal Learning and its Application to Confounded Bandit Policy Evaluationcited by 40
  5. 2021Causal Reinforcement Learning using Observational and Interventional Datacited by 39
  6. 2018A confounding bridge approach for double negative control inference on causal effectscited by 36
  7. 2020Provably Efficient Neural Estimation of Structural Equation Model: An Adversarial Approachcited by 31
  8. 2021Off-Policy Evaluation in Partially Observed Markov Decision Processes under Sequential Ignorabilitycited by 24
  9. 2020The Variational Method of Momentscited by 23
  10. 2021Structured World Belief for Reinforcement Learning in POMDPcited by 21
  11. 2021A Spectral Approach to Off-Policy Evaluation for POMDPscited by 21
  12. 2021Minimax Kernel Machine Learning for a Class of Doubly Robust Functionals with Application to Proximal Causal Inferencecited by 20
  13. 2021Proximal Causal Inference for Complex Longitudinal Studiescited by 16
  14. 2018Learning in POMDPs with Monte Carlo Tree Searchcited by 15
  15. 2020Counterfactually Guided Off-policy Transfer in Clinical Settingscited by 14
  16. 2021Estimating and Improving Dynamic Treatment Regimes With a Time-Varying Instrumental Variablecited by 13
  17. 2020Reinforcement Learning for POMDP: Partitioned Rollout and Policy Iteration with Application to Autonomous Sequential Repair Problemscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 14; 2023: 6; 2024: 2; 2025: 11; 2026: 6.

41 papers.

Sort:

By area:

  1. 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 95
  2. 2023A Survey on Causal Reinforcement Learningcited by 45
  3. 2021A Minimax Learning Approach to Off-Policy Evaluation in Confounded Partially Observable Markov Decision Processescited by 36
  4. 2022Provably Efficient Reinforcement Learning in Partially Observable Dynamical Systemscited by 35
  5. 2022Off-Policy Confidence Interval Estimation with Confounded Markov Decision Processcited by 31
  6. 2022Long-term Causal Inference Under Persistent Confounding via Data Combinationcited by 27
  7. 2022Pessimism in the Face of Confounders: Provably Efficient Offline Reinforcement Learning in Partially Observable Markov Decision Processescited by 26
  8. 2022Future-Dependent Value-Based Off-Policy Evaluation in POMDPscited by 21
  9. 2022An Instrumental Variable Approach to Confounded Off-Policy Evaluationcited by 20
  10. 2022Off-Policy Evaluation for Episodic Partially Observable Markov Decision Processes under Non-Parametric Modelscited by 18
  11. 2022Reinforcement Learning from Partial Observation: Linear Function Approximation with Provable Sample Efficiencycited by 16
  12. 2025Offline Reinforcement Learning in Large State Spaces: Algorithms and Guaranteescited by 10
29 more
  1. 2022Strategic Decision-Making in the Presence of Information Asymmetry: Provably Efficient RL with Algorithmic Instrumentscited by 9
  2. 2023Source Condition Double Robust Inference on Functionals of Inverse Problemscited by 9
  3. 2025Statistical Tractability of Off-policy Evaluation of History-dependent Policies in POMDPscited by 8
  4. 2023Proximal Causal Learning of Conditional Average Treatment Effectscited by 7
  5. 2023Delphic Offline Reinforcement Learning under Nonidentifiable Hidden Confoundingcited by 7
  6. 2025Confounding Robust Deep Reinforcement Learning: A Causal Approachcited by 7
  7. 2022Reinforcement Learning in Modern Biostatistics: Constructing Optimal Adaptive Interventionscited by 6
  8. 2023A Policy Gradient Method for Confounded POMDPscited by 6
  9. 2024RL in Latent MDPs is Tractable: Online Guarantees via Off-Policy Evaluationcited by 6
  10. 2025Automatic Reward Shaping from Confounded Offline Datacited by 6
  11. 2022Optimal Treatment Regimes for Proximal Causal Learningcited by 5
  12. 2024Two-way Deconfounder for Off-policy Evaluation in Causal Reinforcement Learningcited by 5
  13. 2025Counterfactually Fair Reinforcement Learning via Sequential Data Preprocessingcited by 5
  14. 2021Pessimistic Model Selection for Offline Deep Reinforcement Learningcited by 3
  15. 2022Blessing from Human-AI Interaction: Super Reinforcement Learning in Confounded Environmentscited by 3
  16. 2025Reinforcement Learning with Continuous Actions Under Unmeasured Confoundingcited by 3
  17. 2023A Unified Framework of Policy Learning for Contextual Bandit with Confounding Bias and Missing Observationscited by 2
  18. 2025Multi-Agent Reinforcement Learning in Intelligent Transportation Systems: A Comprehensive Surveycited by 2
  19. 2026Causal Flow Q-Learning for Robust Offline Reinforcement Learningcited by 2
  20. 2025Off-Policy Evaluation for Sequential Persuasion Process with Unobserved Confoundingcited by 1
  21. 2025Demystifying the Paradox of Importance Sampling with an Estimated History-Dependent Behavior Policy in Off-Policy Evaluationcited by 1
  22. 2025Model-Based Reinforcement Learning Under Confoundingcited by 1
  23. 2026Design Experiments to Compare Multi-armed Bandit Algorithmscited by 1
  24. 2022Dynamic Structure Estimation from Bandit Feedback using Nonvanishing Exponential Sumscited by 0
  25. 2025The Sample Complexity of Online Strategic Decision Making with Information Asymmetry and Knowledge Transportabilitycited by 0
  26. 2026Causal Imitation Learning Under Measurement Error and Distribution Shiftcited by 0
  27. 2026Cross-fitted Proximal Learning for Model-Based Reinforcement Learningcited by 0
  28. 2026Mind the Sim-to-Real Gap & Think Like a Scientistcited by 0
  29. 2026Off-Policy Evaluation for Missingness-Aware Policies in MDPs with Rewards Missing Not at Randomcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Projected State-action Balancing Weights for Offline Reinforcement Learning
  2. 2024Rethinking State Disentanglement in Causal Reinforcement Learning
  3. 2022Offline Reinforcement Learning with Instrumental Variables in Confounded Markov Decision Processes
  4. 2026Learning from the Unseen: Offline Reinforcement Learning with Hidden Actions
  5. 2022Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Belief
  6. 2026Model-based Bootstrap for Offline Policy Evaluation in Tabular Reinforcement Learning
  7. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning
  8. 2022Offline Policy Evaluation and Optimization under Confounding