mlsanityData to 7 Oct 2026

Causal Deep Reinforcement Learning Using Observational Data

arXiv 2211.15355 ↗First posted 28 Nov 2022cs.LG stat.MLcited by 3

Not on the map. Why?

Abstract (arXiv)

Deep reinforcement learning (DRL) requires the collection of interventional data, which is sometimes expensive and even unethical in the real world, such as in the autonomous driving and the medical field. Offline reinforcement learning promises to alleviate this issue by exploiting the vast amount of observational data available in the real world. However, observational data may mislead the learning agent to undesirable outcomes if the behavior policy that generates the data depends on unobserved random variables (i.e., confounders). In this paper, we propose two deconfounding methods in DRL to address this problem. The methods first calculate the importance degree of different samples based on the causal inference technique, and then adjust the impact of different samples on the loss function by reweighting or resampling the offline dataset to ensure its unbiasedness. These deconfounding methods can be flexibly combined with existing model-free DRL algorithms such as soft actor-critic and deep Q-learning, provided that a weak condition can be satisfied by the loss functions of these algorithms. We prove the effectiveness of our deconfounding methods and validate them experimentally.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 29 of its 49 references that are in the map. First posted on arXiv 28 Nov 2022.

Builds on

29 of 49 references.

References · 29

  1. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  3. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  4. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  5. 2016OpenAI Gymcited by 1,684
  6. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  7. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  8. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  9. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 514
  10. 2017Causal Effect Inference with Deep Latent-Variable Modelscited by 408
  11. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  12. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 226
17 more
  1. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learningcited by 213
  2. 2019Reinforcement Learning in Feature Space: Matrix Bandit, Kernels, and Regret Boundcited by 199
  3. 2016Causal Bandits: Learning Good Interventions via Causal Inferencecited by 100
  4. 2019Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learningcited by 75
  5. 2019Off-Policy Evaluation in Partially Observable Environmentscited by 65
  6. 2018Policy Evaluation and Optimization with Continuous Treatmentscited by 57
  7. 2020Confounding-Robust Policy Evaluation in Infinite-Horizon Reinforcement Learningcited by 56
  8. 2017Consistent On-Line Off-Policy Evaluationcited by 51
  9. 2019Off-Policy Deep Reinforcement Learning by Bootstrapping the Covariate Shiftcited by 47
  10. 2020Provably Efficient Causal Reinforcement Learning with Confounded Observational Datacited by 40
  11. 2017Identifying Best Interventions through Online Importance Samplingcited by 39
  12. 2021Causal Reinforcement Learning using Observational and Interventional Datacited by 38
  13. 2019Conditional Density Estimation with Neural Networks: Best Practices and Benchmarkscited by 37
  14. 2018Deconfounding Reinforcement Learning in Observational Settingscited by 36
  15. 2021A Critical Look at the Consistency of Causal Estimation With Deep Latent Variable Modelscited by 29
  16. 2021A Spectral Approach to Off-Policy Evaluation for POMDPscited by 20
  17. 2017A generic approach to nonparametric function estimation with mixed datacited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 1; 2025: 0; 2026: 1.

3 papers.

  1. 2023Causal Deep Learningcited by 20
  2. 2024BECAUSE: Bilinear Causal Representation for Generalizable Offline Model-based Reinforcement Learningcited by 3
  3. 2026A Closer Look at the Application of Causal Inference in Graph Representation Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Delphic Offline Reinforcement Learning under Nonidentifiable Hidden Confounding
  2. 2021Instrumental Variable Value Iteration for Causal Offline Reinforcement Learning
  3. 2021A Dataset Perspective on Offline Reinforcement Learning
  4. 2025Expert or not? assessing data quality in offline reinforcement learning
  5. 2026Learning Optimal and Sample-Efficient Decision Policies with Guarantees
  6. 2023Can Active Sampling Reduce Causal Confusion in Offline Reinforcement Learning?
  7. 2022Offline Reinforcement Learning with Instrumental Variables in Confounded Markov Decision Processes
  8. 2024Pessimistic Causal Reinforcement Learning with Mediators for Confounded Offline Data