# Estimating and Improving Dynamic Treatment Regimes With a Time-Varying Instrumental Variable

arXiv: https://arxiv.org/abs/2104.07822 · HTML: https://mlsanity.com/paper/2104.07822/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 86 references that are in the map.
- First posted on arXiv 15 Apr 2021.

## Builds on (most cited first, 17 of 17)

- [Contextual Decision Processes with Low Bellman Rank are PAC-Learnable](https://mlsanity.com/paper/1610.09512/) (2016)
- [Information-Theoretic Considerations in Batch Reinforcement Learning](https://mlsanity.com/paper/1905.00360/) (2019)
- [Bilinear Classes: A Structural Framework for Provable Generalization in RL](https://mlsanity.com/paper/2103.10897/) (2021)
- [Confounding-Robust Policy Improvement](https://mlsanity.com/paper/1805.08593/) (2018)
- [Confounding-Robust Policy Evaluation in Infinite-Horizon Reinforcement Learning](https://mlsanity.com/paper/2002.04518/) (2020)
- [Interval Estimation of Individual-Level Causal Effects Under Unobserved Confounding](https://mlsanity.com/paper/1810.02894/) (2018)
- [Residual Weighted Learning for Estimating Individualized Treatment Rules](https://mlsanity.com/paper/1508.03179/) (2015)
- [Instrumental Variable Value Iteration for Causal Offline Reinforcement Learning](https://mlsanity.com/paper/2102.09907/) (2021)
- [Bounded, efficient and multiply robust estimation of average treatment effects using instrumental variables](https://mlsanity.com/paper/1611.09925/) (2016)
- [A semiparametric instrumental variable approach to optimal treatment regimes under endogeneity](https://mlsanity.com/paper/1911.09260/) (2019)
- [Interpretable Dynamic Treatment Regimes](https://mlsanity.com/paper/1606.01472/) (2016)
- [Estimating Optimal Treatment Rules with an Instrumental Variable: A Partial Identification Learning Approach](https://mlsanity.com/paper/2002.02579/) (2020)
- [Sensitivity analysis via the proportion of unmeasured confounding](https://mlsanity.com/paper/1912.02793/) (2019)
- [Decision trees as partitioning machines to characterize their generalization properties](https://mlsanity.com/paper/2010.07374/) (2020)
- [Efficient augmentation and relaxation learning for individualized treatment rules using observational data](https://mlsanity.com/paper/1901.00663/) (2019)
- [Selecting and ranking individualized treatment rules with unmeasured confounding](https://mlsanity.com/paper/2002.10436/) (2020)
- [Sharpening the Rosenbaum Sensitivity Bounds to Address Concerns About Interactions Between Observed and Unobserved Covariates](https://mlsanity.com/paper/2004.00766/) (2020)

## Built on by (most cited first, 13 of 13)

- [A Survey on Causal Reinforcement Learning](https://mlsanity.com/paper/2302.05209/) (2023)
- [Proximal Reinforcement Learning: Efficient Off-Policy Evaluation in Partially Observed Markov Decision Processes](https://mlsanity.com/paper/2110.15332/) (2021)
- [An Instrumental Variable Approach to Confounded Off-Policy Evaluation](https://mlsanity.com/paper/2212.14468/) (2022)
- [Offline Reinforcement Learning with Instrumental Variables in Confounded Markov Decision Processes](https://mlsanity.com/paper/2209.08666/) (2022)
- [Robust Fitted-Q-Evaluation and Iteration under Sequentially Exogenous Unobserved Confounders](https://mlsanity.com/paper/2302.00662/) (2023)
- [Adaptive Discretization in Online Reinforcement Learning](https://mlsanity.com/paper/2110.15843/) (2021)
- [Strategic Decision-Making in the Presence of Information Asymmetry: Provably Efficient RL with Algorithmic Instruments](https://mlsanity.com/paper/2208.11040/) (2022)
- [Blessing from Human-AI Interaction: Super Reinforcement Learning in Confounded Environments](https://mlsanity.com/paper/2209.15448/) (2022)
- [A Unified Framework of Policy Learning for Contextual Bandit with Confounding Bias and Missing Observations](https://mlsanity.com/paper/2303.11187/) (2023)
- [Statistical Estimation of Confounded Linear MDPs: An Instrumental Variable Approach](https://mlsanity.com/paper/2209.05186/) (2022)
- [Offline Reinforcement Learning for Human-Guided Human-Machine Interaction with Private Information](https://mlsanity.com/paper/2212.12167/) (2022)
- [Pessimistic Causal Reinforcement Learning with Mediators for Confounded Offline Data](https://mlsanity.com/paper/2403.11841/) (2024)
- [The Sample Complexity of Online Strategic Decision Making with Information Asymmetry and Knowledge Transportability](https://mlsanity.com/paper/2506.09940/) (2025)

## Abstract (arXiv)

Estimating dynamic treatment regimes (DTRs) from retrospective observational data is challenging as some degree of unmeasured confounding is often expected. In this work, we develop a framework of estimating properly defined "optimal" DTRs with a time-varying instrumental variable (IV) when unmeasured covariates confound the treatment and outcome, rendering the potential outcome distributions only partially identified. We derive a novel Bellman equation under partial identification, use it to define a generic class of estimands (termed IV-optimal DTRs), and study the associated estimation problem. We then extend the IV-optimality framework to tackle the policy improvement problem, delivering IV-improved DTRs that are guaranteed to perform no worse and potentially better than a pre-specified baseline DTR. Importantly, our IV-improvement framework opens up the possibility of strictly improving upon DTRs that are optimal under the no unmeasured confounding assumption (NUCA). We demonstrate via extensive simulations the superior performance of IV-optimal and IV-improved DTRs over the DTRs that are optimal only under the NUCA. In a real data example, we embed retrospective observational registry data into a natural, two-stage experiment with noncompliance using a time-varying IV and estimate useful IV-optimal DTRs that assign mothers to high-level or low-level neonatal intensive care units based on their prognostic variables.