mlsanityData to 7 Oct 2026

Provably Efficient Causal Reinforcement Learning with Confounded Observational Data

arXiv 2006.12311 ↗First posted 22 Jun 2020cs.LG stat.MLcited by 40

Not on the map. Why?

Abstract (arXiv)

Empowered by expressive function approximators such as neural networks, deep reinforcement learning (DRL) achieves tremendous empirical successes. However, learning expressive function approximators requires collecting a large dataset (interventional data) by interacting with the environment. Such a lack of sample efficiency prohibits the application of DRL to critical scenarios, e.g., autonomous driving and personalized medicine, since trial and error in the online setting is often unsafe and even unethical. In this paper, we study how to incorporate the dataset (observational data) collected offline, which is often abundantly available in practice, to improve the sample efficiency in the online setting. To incorporate the possibly confounded observational data, we propose the deconfounded optimistic value iteration (DOVI) algorithm, which incorporates the confounded observational data in a provably efficient manner. More specifically, DOVI explicitly adjusts for the confounding bias in the observational data, where the confounders are partially observed or unobserved. In both cases, such adjustments allow us to construct the bonus based on a notion of information gain, which takes into account the amount of information acquired from the offline setting. In particular, we prove that the regret of DOVI is smaller than the optimal regret achievable in the pure online setting by a multiplicative factor, which decreases towards zero when the confounded observational data are more informative upon the adjustments. Our algorithm and analysis serve as a step towards causal reinforcement learning.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 11; 2023: 9; 2024: 8; 2025: 6; 2026: 2.

40 papers.

Sort:

By area:

  1. 2023A Survey on Causal Reinforcement Learningcited by 43
  2. 2021Proximal Reinforcement Learning: Efficient Off-Policy Evaluation in Partially Observed Markov Decision Processescited by 40
  3. 2021A Minimax Learning Approach to Off-Policy Evaluation in Confounded Partially Observable Markov Decision Processescited by 34
  4. 2022Off-Policy Confidence Interval Estimation with Confounded Markov Decision Processcited by 31
  5. 2023Seeing is not Believing: Robust Reinforcement Learning against Spurious Correlationcited by 25
  6. 2023Explainable Reinforcement Learning via a Causal World Modelcited by 22
  7. 2023Causal Reinforcement Learning: A Surveycited by 22
  8. 2021Dynamic Bottleneck for Robust Self-Supervised Explorationcited by 21
  9. 2022Future-Dependent Value-Based Off-Policy Evaluation in POMDPscited by 20
  10. 2022An Instrumental Variable Approach to Confounded Off-Policy Evaluationcited by 20
  11. 2023Causal Deep Learningcited by 20
  12. 2022A Relational Intervention Approach for Unsupervised Dynamics Generalization in Model-Based Reinforcement Learningcited by 15
28 more
  1. 2021On Covariate Shift of Latent Confounders in Imitation and Reinforcement Learningcited by 14
  2. 2022Offline Policy Evaluation and Optimization under Confoundingcited by 12
  3. 2023Robust Fitted-Q-Evaluation and Iteration under Sequentially Exogenous Unobserved Confounderscited by 12
  4. 2023Finding Counterfactually Optimal Action Sequences in Continuous State Spacescited by 12
  5. 2022Strategic Decision-Making in the Presence of Information Asymmetry: Provably Efficient RL with Algorithmic Instrumentscited by 9
  6. 2023Delphic Offline Reinforcement Learning under Nonidentifiable Hidden Confoundingcited by 7
  7. 2024Regularized DeepIV with Model Selectioncited by 6
  8. 2024A Natural Extension To Online Algorithms For Hybrid RL With Limited Coveragecited by 5
  9. 2024Two-way Deconfounder for Off-policy Evaluation in Causal Reinforcement Learningcited by 5
  10. 2025Efficient and Sharp Off-Policy Learning under Unobserved Confoundingcited by 5
  11. 2022A Multi-Agent Reinforcement Learning Framework for Off-Policy Evaluation in Two-sided Marketscited by 4
  12. 2023Offline Recommender System Evaluation under Unobserved Confoundingcited by 4
  13. 2025Improve the Training Efficiency of DRL for Wireless Communication Resource Allocation: The Role of Generative Diffusion Modelscited by 4
  14. 2022Blessing from Human-AI Interaction: Super Reinforcement Learning in Confounded Environmentscited by 3
  15. 2022Causal Deep Reinforcement Learning Using Observational Datacited by 3
  16. 2024BECAUSE: Bilinear Causal Representation for Generalizable Offline Model-based Reinforcement Learningcited by 3
  17. 2025Expediting Reinforcement Learning by Incorporating Knowledge About Temporal Causality in the Environmentcited by 2
  18. 2024Why Online Reinforcement Learning is Causalcited by 1
  19. 2022Statistical Estimation of Confounded Linear MDPs: An Instrumental Variable Approachcited by 0
  20. 2022Offline Reinforcement Learning for Human-Guided Human-Machine Interaction with Private Informationcited by 0
  21. 2024Pessimistic Causal Reinforcement Learning with Mediators for Confounded Offline Datacited by 0
  22. 2024Disentangled Representations for Causal Cognitioncited by 0
  23. 2024Benchmarks for Reinforcement Learning with Biased Offline Data and Imperfect Simulatorscited by 0
  24. 2025Causal Policy Learning in Reinforcement Learning: Backdoor-Adjusted Soft Actor-Criticcited by 0
  25. 2025The Sample Complexity of Online Strategic Decision Making with Information Asymmetry and Knowledge Transportabilitycited by 0
  26. 2025Causal Reinforcement Learning based Agent-Patient Interaction with Clinical Domain Knowledgecited by 0
  27. 2026CausalGDP: Causality-Guided Diffusion Policies for Reinforcement Learningcited by 0
  28. 2026Quantifying Potential Observation Missingness in Inverse Reinforcement Learningcited by 0