mlsanityData to 14 Oct 2026

Offline Reinforcement Learning with Instrumental Variables in Confounded Markov Decision Processes

arXiv 2209.08666 ↗First posted 18 Sep 2022cs.LG stat.MEcited by 19

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

We study the offline reinforcement learning (RL) in the face of unmeasured confounders. Due to the lack of online interaction with the environment, offline RL is facing the following two significant challenges: (i) the agent may be confounded by the unobserved state variables; (ii) the offline data collected a prior does not provide sufficient coverage for the environment. To tackle the above challenges, we study the policy learning in the confounded MDPs with the aid of instrumental variables. Specifically, we first establish value function (VF)-based and marginalized importance sampling (MIS)-based identification results for the expected total reward in the confounded MDPs. Then by leveraging pessimism and our identification results, we propose various policy learning methods with the finite-sample suboptimality guarantee of finding the optimal in-class policy under minimal data coverage and modeling assumptions. Lastly, our extensive theoretical investigations and one numerical study motivated by the kidney transplantation demonstrate the promising performance of the proposed methods.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 26 of its 79 references that are in the map. First posted on arXiv 18 Sep 2022.

Builds on

26 of 79 references.

Closest ancestors · 12

Same direction, newest first

  1. 2022Pessimism in the Face of Confounders: Provably Efficient Offline Reinforcement Learning in Partially Observable Markov Decision Processes26 May
  2. 2022Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexity28 Feb
  3. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrability9 Feb
  4. 2021False Correlation Reduction for Offline Reinforcement Learning24 Oct
  5. 2021Projected State-action Balancing Weights for Offline Reinforcement Learning10 Sep
  6. 2021Instrumental Variable Value Iteration for Causal Offline Reinforcement Learning19 Feb
  7. 2021Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiency5 Feb
  8. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?22 Oct
  9. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  10. 2020MOPO: Model-based Offline Policy Optimization27 May
  11. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  12. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May

Other references · 14

  1. 2016Safe, Multi-Agent, Reinforcement Learning for Autonomous Drivingcited by 311
  2. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 287
  3. 2020GenDICE: Generalized Offline Estimation of Stationary Valuescited by 122
  4. 2020An Introduction to Proximal Causal Learningcited by 95
  5. 2020Batch Policy Learning in Average Reward Markov Decision Processescited by 69
  6. 2020Statistical Inference of the Value Function for Reinforcement Learning in Infinite Horizon Settingscited by 65
  7. 2019Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimationcited by 48
  8. 2022The Efficacy of Pessimism in Asynchronous Q-Learningcited by 42
  9. 2021A Minimax Learning Approach to Off-Policy Evaluation in Confounded Partially Observable Markov Decision Processescited by 36
  10. 2022On Well-posedness and Minimax Optimal Rates of Nonparametric Q-function Estimation in Off-policy Evaluationcited by 33
  11. 2022Off-Policy Confidence Interval Estimation with Confounded Markov Decision Processcited by 31
  12. 2021Estimating Optimal Infinite Horizon Dynamic Treatment Regimes via pT-Learningcited by 26
2 more
  1. 2021Estimating and Improving Dynamic Treatment Regimes With a Time-Varying Instrumental Variablecited by 13
  2. 2021Generalization bounds for nonparametric regression with β-mixing samplescited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 6; 2023: 4; 2024: 6; 2025: 2; 2026: 1.

19 papers.

Sort:

By area:

  1. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
  2. 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 95
  3. 2022An Instrumental Variable Approach to Confounded Off-Policy Evaluationcited by 20
  4. 2024Functional Bilevel Optimization for Machine Learningcited by 20
  5. 2023Robust Fitted-Q-Evaluation and Iteration under Sequentially Exogenous Unobserved Confounderscited by 13
  6. 2022Offline Policy Evaluation and Optimization under Confoundingcited by 12
  7. 2024Low-Rank Contextual Reinforcement Learning from Heterogeneous Human Feedbackcited by 10
  8. 2022Optimizing Pessimism in Dynamic Treatment Regimes: A Bayesian Learning Approachcited by 7
  9. 2023Robust Offline Reinforcement learning with Heavy-Tailed Rewardscited by 7
  10. 2024Learning Decision Policies with Instrumental Variables through Double Machine Learningcited by 5
  11. 2024Two-way Deconfounder for Off-policy Evaluation in Causal Reinforcement Learningcited by 5
  12. 2022Blessing from Human-AI Interaction: Super Reinforcement Learning in Confounded Environmentscited by 3
7 more
  1. 2023STEEL: Singularity-aware Reinforcement Learningcited by 3
  2. 2023PASTA: Pessimistic Assortment Optimizationcited by 3
  3. 2025Reinforcement Learning with Continuous Actions Under Unmeasured Confoundingcited by 3
  4. 2025PASTA: A Unified Framework for Offline Assortment Learningcited by 2
  5. 2022Offline Reinforcement Learning for Human-Guided Human-Machine Interaction with Private Informationcited by 0
  6. 2024Pessimistic Causal Reinforcement Learning with Mediators for Confounded Offline Datacited by 0
  7. 2026Exact Distinguishability in Non-Markovian Decision Processescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Model-Based Reinforcement Learning Under Confounding
  2. 2023Delphic Offline Reinforcement Learning under Nonidentifiable Hidden Confounding
  3. 2022Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Belief
  4. 2023On the Sample Complexity of Vanilla Model-Based Offline Reinforcement Learning with Dependent Samples
  5. 2021Proximal Reinforcement Learning: Efficient Off-Policy Evaluation in Partially Observed Markov Decision Processes
  6. 2022Offline Reinforcement Learning Under Value and Density-Ratio Realizability: The Power of Gaps
  7. 2022Optimal Conservative Offline RL with General Function Approximation via Augmented Lagrangian
  8. 2025Semi-pessimistic Reinforcement Learning