# Asymptotic Theory for IV-Based Reinforcement Learning with Potential Endogeneity

arXiv: https://arxiv.org/abs/2103.04021 · HTML: https://mlsanity.com/paper/2103.04021/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Mar 2021.

## Built on by (most cited first, 8 of 8)

- [On Instrumental Variable Regression for Deep Offline Policy Evaluation](https://mlsanity.com/paper/2105.10148/) (2021)
- [An Instrumental Variable Approach to Confounded Off-Policy Evaluation](https://mlsanity.com/paper/2212.14468/) (2022)
- [Causal Multi-Agent Reinforcement Learning: Review and Open Problems](https://mlsanity.com/paper/2111.06721/) (2021)
- [Spectral Representation Learning for Conditional Moment Models](https://mlsanity.com/paper/2210.16525/) (2022)
- [Instrumental Variables in Causal Inference and Machine Learning: A Survey](https://mlsanity.com/paper/2212.05778/) (2022)
- [Dynamic Selection in Algorithmic Decision-making](https://mlsanity.com/paper/2108.12547/) (2021)
- [Statistical Estimation of Confounded Linear MDPs: An Instrumental Variable Approach](https://mlsanity.com/paper/2209.05186/) (2022)
- [Pessimistic Causal Reinforcement Learning with Mediators for Confounded Offline Data](https://mlsanity.com/paper/2403.11841/) (2024)

## Abstract (arXiv)

In the standard data analysis framework, data is collected (once and for all), and then data analysis is carried out. However, with the advancement of digital technology, decision-makers constantly analyze past data and generate new data through their decisions. We model this as a Markov decision process and show that the dynamic interaction between data generation and data analysis leads to a new type of bias -- reinforcement bias -- that exacerbates the endogeneity problem in standard data analysis. We propose a class of instrument variable (IV)-based reinforcement learning (RL) algorithms to correct for the bias and establish their theoretical properties by incorporating them into a stochastic approximation (SA) framework. Our analysis accommodates iterate-dependent Markovian structures and, therefore, can be used to study RL algorithms with policy improvement. We also provide formulas for inference on optimal policies of the IV-RL algorithms. These formulas highlight how intertemporal dependency of the Markovian environment affects the inference.