mlsanityData to 7 Oct 2026

Estimating and Improving Dynamic Treatment Regimes With a Time-Varying Instrumental Variable

arXiv 2104.07822 ↗First posted 15 Apr 2021stat.ME cs.LG econ.EMcited by 13

Not on the map. Why?

Abstract (arXiv)

Estimating dynamic treatment regimes (DTRs) from retrospective observational data is challenging as some degree of unmeasured confounding is often expected. In this work, we develop a framework of estimating properly defined "optimal" DTRs with a time-varying instrumental variable (IV) when unmeasured covariates confound the treatment and outcome, rendering the potential outcome distributions only partially identified. We derive a novel Bellman equation under partial identification, use it to define a generic class of estimands (termed IV-optimal DTRs), and study the associated estimation problem. We then extend the IV-optimality framework to tackle the policy improvement problem, delivering IV-improved DTRs that are guaranteed to perform no worse and potentially better than a pre-specified baseline DTR. Importantly, our IV-improvement framework opens up the possibility of strictly improving upon DTRs that are optimal under the no unmeasured confounding assumption (NUCA). We demonstrate via extensive simulations the superior performance of IV-optimal and IV-improved DTRs over the DTRs that are optimal only under the NUCA. In a real data example, we embed retrospective observational registry data into a natural, two-stage experiment with noncompliance using a time-varying IV and estimate useful IV-optimal DTRs that assign mothers to high-level or low-level neonatal intensive care units based on their prognostic variables.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 17 of its 86 references that are in the map. First posted on arXiv 15 Apr 2021.

Builds on

17 of 86 references.

References · 17

  1. 2016Contextual Decision Processes with Low Bellman Rank are PAC-Learnablecited by 290
  2. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
  3. 2021Bilinear Classes: A Structural Framework for Provable Generalization in RLcited by 176
  4. 2018Confounding-Robust Policy Improvementcited by 73
  5. 2020Confounding-Robust Policy Evaluation in Infinite-Horizon Reinforcement Learningcited by 56
  6. 2018Interval Estimation of Individual-Level Causal Effects Under Unobserved Confoundingcited by 51
  7. 2015Residual Weighted Learning for Estimating Individualized Treatment Rulescited by 37
  8. 2021Instrumental Variable Value Iteration for Causal Offline Reinforcement Learningcited by 35
  9. 2016Bounded, efficient and multiply robust estimation of average treatment effects using instrumental variablescited by 23
  10. 2019A semiparametric instrumental variable approach to optimal treatment regimes under endogeneitycited by 22
  11. 2016Interpretable Dynamic Treatment Regimescited by 17
  12. 2020Estimating Optimal Treatment Rules with an Instrumental Variable: A Partial Identification Learning Approachcited by 17
5 more
  1. 2019Sensitivity analysis via the proportion of unmeasured confoundingcited by 13
  2. 2020Decision trees as partitioning machines to characterize their generalization propertiescited by 12
  3. 2019Efficient augmentation and relaxation learning for individualized treatment rules using observational datacited by 9
  4. 2020Selecting and ranking individualized treatment rules with unmeasured confoundingcited by 4
  5. 2020Sharpening the Rosenbaum Sensitivity Bounds to Address Concerns About Interactions Between Observed and Unobserved Covariatescited by 3

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 6; 2023: 3; 2024: 1; 2025: 1.

13 papers.

Sort:

By area:

  1. 2023A Survey on Causal Reinforcement Learningcited by 43
  2. 2021Proximal Reinforcement Learning: Efficient Off-Policy Evaluation in Partially Observed Markov Decision Processescited by 40
  3. 2022An Instrumental Variable Approach to Confounded Off-Policy Evaluationcited by 20
  4. 2022Offline Reinforcement Learning with Instrumental Variables in Confounded Markov Decision Processescited by 19
  5. 2021Adaptive Discretization in Online Reinforcement Learningcited by 12
  6. 2023Robust Fitted-Q-Evaluation and Iteration under Sequentially Exogenous Unobserved Confounderscited by 12
  7. 2022Strategic Decision-Making in the Presence of Information Asymmetry: Provably Efficient RL with Algorithmic Instrumentscited by 9
  8. 2022Blessing from Human-AI Interaction: Super Reinforcement Learning in Confounded Environmentscited by 3
  9. 2023A Unified Framework of Policy Learning for Contextual Bandit with Confounding Bias and Missing Observationscited by 2
  10. 2022Statistical Estimation of Confounded Linear MDPs: An Instrumental Variable Approachcited by 0
  11. 2022Offline Reinforcement Learning for Human-Guided Human-Machine Interaction with Private Informationcited by 0
  12. 2024Pessimistic Causal Reinforcement Learning with Mediators for Confounded Offline Datacited by 0
1 more
  1. 2025The Sample Complexity of Online Strategic Decision Making with Information Asymmetry and Knowledge Transportabilitycited by 0