Statistical Estimation of Confounded Linear MDPs: An Instrumental Variable Approach
Not on the map. Why?
Abstract (arXiv)
In an Markov decision process (MDP), unobservable confounders may exist and have impacts on the data generating process, so that the classic off-policy evaluation (OPE) estimators may fail to identify the true value function of the target policy. In this paper, we study the statistical properties of OPE in confounded MDPs with observable instrumental variables. Specifically, we propose a two-stage estimator based on the instrumental variables and establish its statistical properties in the confounded MDPs with a linear structure. For non-asymptotic analysis, we prove a O(n-1/2)-error bound where n is the number of samples. For asymptotic analysis, we prove that the two-stage estimator is asymptotically normal with a typical rate of n1/2. To the best of our knowledge, we are the first to show such statistical results of the two-stage estimator for confounded linear MDPs via instrumental variables.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 28 of its 57 references that are in the map. First posted on arXiv 12 Sep 2022.
Builds on
28 of 57 references.
References · 28
- 2015Continuous control with deep reinforcement learningcited by 3,407
- 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
- 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
- 2020MOReL : Model-Based Offline Reinforcement Learningcited by 590
- 2020Is Pessimism Provably Efficient for Offline RL?cited by 364
- 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 287
- 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?cited by 136
- 2021Provably Efficient Reinforcement Learning with Linear Function Approximation Under Adaptivity Constraintscited by 124
- 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrabilitycited by 123
- 2020Minimax-Optimal Off-Policy Evaluation with Linear Function Approximationcited by 120
- 2019Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processescited by 119
- 2019Off-Policy Evaluation in Partially Observable Environmentscited by 66
16 more
- 2021Risk Bounds and Rademacher Complexity in Batch Reinforcement Learningcited by 42
- 2020Off-policy Evaluation in Infinite-Horizon Reinforcement Learning with Latent Confounderscited by 41
- 2020Provably Efficient Causal Reinforcement Learning with Confounded Observational Datacited by 40
- 2021Deep Proxy Causal Learning and its Application to Confounded Bandit Policy Evaluationcited by 40
- 2018Deconfounding Reinforcement Learning in Observational Settingscited by 36
- 2021Instrumental Variable Value Iteration for Causal Offline Reinforcement Learningcited by 35
- 2021Variance-Aware Off-Policy Evaluation with Linear Function Approximationcited by 34
- 2022Pessimism in the Face of Confounders: Provably Efficient Offline Reinforcement Learning in Partially Observable Markov Decision Processescited by 26
- 2019A semiparametric instrumental variable approach to optimal treatment regimes under endogeneitycited by 22
- 2020Q* Approximation Schemes for Batch Reinforcement Learning: A Theoretical Comparisoncited by 22
- 2021On Instrumental Variable Regression for Deep Offline Policy Evaluationcited by 22
- 2018Semiparametric Contextual Banditscited by 19
- 2021Estimating and Improving Dynamic Treatment Regimes With a Time-Varying Instrumental Variablecited by 13
- 2021Asymptotic Theory for IV-Based Reinforcement Learning with Potential Endogeneitycited by 8
- 2022Adaptive Experimentation in the Presence of Exogenous Nonstationary Variationcited by 8
- 2021Dynamic Selection in Algorithmic Decision-makingcited by 1
Built on by
Not cited yet.