Blessing from Human-AI Interaction: Super Reinforcement Learning in Confounded Environments
Not on the map. Why?
Abstract (arXiv)
As AI becomes more prevalent throughout society, effective methods of integrating humans and AI systems that leverage their respective strengths and mitigate risk have become an important priority. In this paper, we introduce the paradigm of super policy learning that takes advantage of Human-AI interaction for data driven sequential decision making. This approach utilizes the observed action, either from AI or humans, as input for achieving a stronger oracle in policy learning for the decision maker (humans or AI). In the decision process with unmeasured confounding, the actions taken by past agents can offer valuable insights into undisclosed information. By including this information for the policy search in a novel and legitimate manner, the proposed super policy learning will yield a super-policy that is guaranteed to outperform both the standard optimal policy and the behavior one (e.g., past agents' actions). We call this stronger oracle a blessing from human-AI interaction. Furthermore, to address the issue of unmeasured confounding in finding super-policies using the batch data, a number of nonparametric and causal identifications are established under the framework of proximal causal inference. Building upon on these novel identification results, we develop several super-policy learning algorithms and systematically study their theoretical properties such as finite-sample regret guarantee. Finally, we illustrate the effectiveness of our proposal through extensive simulations and real-world applications.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 28 of its 63 references that are in the map. First posted on arXiv 29 Sep 2022.
Builds on
28 of 63 references.
References · 28
- 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
- 2016Identifying Causal Effects With Proxy Variables of an Unmeasured Confoundercited by 138
- 2017Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spacescited by 122
- 2017Trial without Error: Towards Safe Reinforcement Learning via Human Interventioncited by 102
- 2020An Introduction to Proximal Causal Learningcited by 91
- 2020Minimax Estimation of Conditional Moment Modelscited by 83
- 2020Sample-Efficient Reinforcement Learning of Undercomplete POMDPscited by 66
- 2019Off-Policy Evaluation in Partially Observable Environmentscited by 65
- 2020Off-policy Policy Evaluation For Sequential Decisions Under Unobserved Confoundingcited by 60
- 2020Confounding-Robust Policy Evaluation in Infinite-Horizon Reinforcement Learningcited by 56
- 2020Provably Efficient Causal Reinforcement Learning with Confounded Observational Datacited by 40
- 2021Proximal Reinforcement Learning: Efficient Off-Policy Evaluation in Partially Observed Markov Decision Processescited by 40
16 more
- 2018A confounding bridge approach for double negative control inference on causal effectscited by 36
- 2021Instrumental Variable Value Iteration for Causal Offline Reinforcement Learningcited by 35
- 2022Provably Efficient Reinforcement Learning in Partially Observable Dynamical Systemscited by 35
- 2022Off-Policy Confidence Interval Estimation with Confounded Markov Decision Processcited by 31
- 2022Pessimism in the Face of Confounders: Provably Efficient Offline Reinforcement Learning in Partially Observable Markov Decision Processescited by 26
- 2021Proximal Learning for Individualized Treatment Regimes Under Unmeasured Confoundingcited by 21
- 2021A Spectral Approach to Off-Policy Evaluation for POMDPscited by 20
- 2022Future-Dependent Value-Based Off-Policy Evaluation in POMDPscited by 20
- 2022Offline Reinforcement Learning with Instrumental Variables in Confounded Markov Decision Processescited by 19
- 2022Off-Policy Evaluation for Episodic Partially Observable Markov Decision Processes under Non-Parametric Modelscited by 18
- 2021Proximal Causal Inference for Complex Longitudinal Studiescited by 16
- 2021Estimating and Improving Dynamic Treatment Regimes With a Time-Varying Instrumental Variablecited by 13
- 2022Model-Free and Model-Based Policy Evaluation when Causality is Uncertaincited by 12
- 2017Learning to Teach Reinforcement Learning Agentscited by 10
- 2022Optimal regimes for algorithm-assisted human decision-makingcited by 9
- 2022Strategic Decision-Making in the Presence of Information Asymmetry: Provably Efficient RL with Algorithmic Instrumentscited by 9
Built on by
Citing papers per year: 2023: 1; 2024: 1; 2025: 0; 2026: 1.
3 papers.