mlsanityData to 7 Oct 2026

Instrumental Variable Value Iteration for Causal Offline Reinforcement Learning

arXiv 2102.09907 ↗First posted 19 Feb 2021stat.ML cs.LGcited by 35

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

In offline reinforcement learning (RL) an optimal policy is learned solely from a priori collected observational data. However, in observational data, actions are often confounded by unobserved variables. Instrumental variables (IVs), in the context of RL, are the variables whose influence on the state variables is all mediated by the action. When a valid instrument is present, we can recover the confounded transition dynamics through observational data. We study a confounded Markov decision process where the transition dynamics admit an additive nonlinear functional form. Using IVs, we derive a conditional moment restriction through which we can identify transition dynamics based on observational data. We propose a provably efficient IV-aided Value Iteration (IVVI) algorithm based on a primal-dual reformulation of the conditional moment restriction. To our knowledge, this is the first provably efficient algorithm for instrument-aided offline RL.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 29 of its 76 references that are in the map. First posted on arXiv 19 Feb 2021.

Builds on

29 of 76 references.

Closest ancestors · 1

Same direction, newest first

  1. 2020Confounding-Robust Policy Evaluation in Infinite-Horizon Reinforcement Learning11 Feb

Other references · 28

  1. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  2. 2019Provably Efficient Exploration in Policy Optimizationcited by 203
  3. 2016Identifying Causal Effects With Proxy Variables of an Unmeasured Confoundercited by 138
  4. 2019Kernel Instrumental Variable Regressioncited by 118
  5. 2019Counterfactual Off-Policy Evaluation with Gumbel-Max Structural Causal Modelscited by 114
  6. 2018SGD and Hogwild! Convergence Without the Bounded Gradients Assumptioncited by 94
  7. 2019Deep Generalized Method of Moments for Instrumental Variable Analysiscited by 88
  8. 2020Information Theoretic Regret Bounds for Online Nonlinear Controlcited by 87
  9. 2020Minimax Estimation of Conditional Moment Modelscited by 83
  10. 2018Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Searchcited by 81
  11. 2020Active Learning for Nonlinear System Identification with Guaranteescited by 78
  12. 2018Confounding-Robust Policy Improvementcited by 73
16 more
  1. 2017Continuous State-Space Models for Optimal Sepsis Treatment - a Deep Reinforcement Learning Approachcited by 68
  2. 2013Counterfactual Probabilities: Computational Methods, Bounds and Applicationscited by 67
  3. 2019Off-Policy Evaluation in Partially Observable Environmentscited by 65
  4. 2020Off-policy Policy Evaluation For Sequential Decisions Under Unobserved Confoundingcited by 60
  5. 2017Stochastic Variance Reduction Methods for Policy Evaluationcited by 56
  6. 2018Interval Estimation of Individual-Level Causal Effects Under Unobserved Confoundingcited by 51
  7. 2017A Reinforcement Learning Approach to Weaning of Mechanical Ventilation in Intensive Care Unitscited by 50
  8. 2020Off-policy Evaluation in Infinite-Horizon Reinforcement Learning with Latent Confounderscited by 40
  9. 2018Deconfounding Reinforcement Learning in Observational Settingscited by 36
  10. 2018Adversarial Generalized Method of Momentscited by 34
  11. 2020Provably Efficient Neural Estimation of Structural Equation Model: An Adversarial Approachcited by 31
  12. 2015Optimal Sup-norm Rates and Uniform Inference on Nonlinear Functionals of Nonparametric IV Regressioncited by 30
  13. 2018Linear Convergence of the Primal-Dual Gradient Method for Convex-Concave Saddle Point Problems without Strong Convexitycited by 28
  14. 2020Estimating Optimal Treatment Rules with an Instrumental Variable: A Partial Identification Learning Approachcited by 17
  15. 2017Exploiting Strong Convexity from Data with Primal-Dual First-Order Algorithmscited by 8
  16. 2019Dual Instrumental Variable Regressioncited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 11; 2023: 7; 2024: 7; 2025: 3; 2026: 2.

35 papers.

Sort:

By area:

  1. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
  2. 2023A Survey on Causal Reinforcement Learningcited by 43
  3. 2021Proximal Reinforcement Learning: Efficient Off-Policy Evaluation in Partially Observed Markov Decision Processescited by 40
  4. 2021A Minimax Learning Approach to Off-Policy Evaluation in Confounded Partially Observable Markov Decision Processescited by 34
  5. 2024Causality for Large Language Modelscited by 24
  6. 2021On Instrumental Variable Regression for Deep Offline Policy Evaluationcited by 22
  7. 2023Causal Reinforcement Learning: A Surveycited by 22
  8. 2022Future-Dependent Value-Based Off-Policy Evaluation in POMDPscited by 20
  9. 2022An Instrumental Variable Approach to Confounded Off-Policy Evaluationcited by 20
  10. 2022Offline Reinforcement Learning with Instrumental Variables in Confounded Markov Decision Processescited by 19
  11. 2022Off-Policy Evaluation for Episodic Partially Observable Markov Decision Processes under Non-Parametric Modelscited by 18
  12. 2023Structure in Deep Reinforcement Learning: A Survey and Open Problemscited by 16
23 more
  1. 2021Estimating and Improving Dynamic Treatment Regimes With a Time-Varying Instrumental Variablecited by 13
  2. 2023Minimax Instrumental Variable Regression and L2 Convergence Guarantees without Identification or Closednesscited by 13
  3. 2022Offline Policy Evaluation and Optimization under Confoundingcited by 12
  4. 2023Robust Fitted-Q-Evaluation and Iteration under Sequentially Exogenous Unobserved Confounderscited by 12
  5. 2022Instrumental Variables in Causal Inference and Machine Learning: A Surveycited by 10
  6. 2023On the Opportunities and Challenges of Offline Reinforcement Learning for Recommender Systemscited by 10
  7. 2022Strategic Decision-Making in the Presence of Information Asymmetry: Provably Efficient RL with Algorithmic Instrumentscited by 9
  8. 2024Nonparametric Instrumental Regression via Kernel Methods is Minimax Optimalcited by 7
  9. 2022Optimal Treatment Regimes for Proximal Causal Learningcited by 5
  10. 2024Learning Decision Policies with Instrumental Variables through Double Machine Learningcited by 5
  11. 2024Two-way Deconfounder for Off-policy Evaluation in Causal Reinforcement Learningcited by 5
  12. 2022Blessing from Human-AI Interaction: Super Reinforcement Learning in Confounded Environmentscited by 3
  13. 2024Learning Causal Dynamics Models in Object-Oriented Environmentscited by 3
  14. 2025Optimality and Adaptivity of Deep Neural Features for Instrumental Variable Regressioncited by 3
  15. 2021Instrument Space Selection for Kernel Maximum Moment Restrictioncited by 2
  16. 2023A Unified Framework of Policy Learning for Contextual Bandit with Confounding Bias and Missing Observationscited by 2
  17. 2025Causality-informed Anomaly Detection in Partially Observable Sensor Networks: Moving beyond Correlationscited by 2
  18. 2022Statistical Estimation of Confounded Linear MDPs: An Instrumental Variable Approachcited by 0
  19. 2022Offline Reinforcement Learning for Human-Guided Human-Machine Interaction with Private Informationcited by 0
  20. 2024Pessimistic Causal Reinforcement Learning with Mediators for Confounded Offline Datacited by 0
  21. 2025The Sample Complexity of Online Strategic Decision Making with Information Asymmetry and Knowledge Transportabilitycited by 0
  22. 2026Adaptive Exploration for Latent-State Banditscited by 0
  23. 2026Identifying Latent Actions and Dynamics from Offline Data via Demonstrator Diversitycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Pessimism in the Face of Confounders: Provably Efficient Offline Reinforcement Learning in Partially Observable Markov Decision Processes
  2. 2024Reward-Relevance-Filtered Linear Offline Reinforcement Learning
  3. 2022Conservative Bayesian Model-Based Value Expansion for Offline Policy Optimization
  4. 2024An Investigation of Offline Reinforcement Learning in Factorisable Action Spaces
  5. 2021Projected State-action Balancing Weights for Offline Reinforcement Learning
  6. 2026Peng's Q(λ) for Conservative Value Estimation in Offline Reinforcement Learning
  7. 2024Compositional Conservatism: A Transductive Approach in Offline Reinforcement Learning
  8. 2022Near-optimal Offline Reinforcement Learning with Linear Representation: Leveraging Variance Information with Pessimism