mlsanityData to 14 Oct 2026

Deconfounding Reinforcement Learning in Observational Settings

arXiv 1812.10576 ↗First posted 26 Dec 2018cs.LG stat.MLcited by 36

Not on the map. Why?

Abstract (arXiv)

We propose a general formulation for addressing reinforcement learning (RL) problems in settings with observational data. That is, we consider the problem of learning good policies solely from historical data in which unobserved factors (confounders) affect both observed actions and rewards. Our formulation allows us to extend a representative RL algorithm, the Actor-Critic method, to its deconfounding variant, with the methodology for this extension being easily applied to other RL algorithms. In addition to this, we develop a new benchmark for evaluating deconfounding RL algorithms by modifying the OpenAI Gym environments and the MNIST dataset. Using this benchmark, we demonstrate that the proposed algorithms are superior to traditional RL methods in confounded environments with observational data. To the best of our knowledge, this is the first time that confounders are taken into consideration for addressing full RL problems with observational data. Code is available at https://github.com/CausalRL/DRL.

Key facts. Cited by 36 papers in the map (data through 14 Oct 2026). First posted on arXiv 26 Dec 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2019: 1; 2020: 0; 2021: 11; 2022: 4; 2023: 6; 2024: 8; 2025: 3; 2026: 3.

36 papers.

Sort:

By area:

  1. 2021Towards Causal Representation Learningcited by 209
  2. 2021Independent mechanism analysis, a new concept?cited by 97
  3. 2021Causal Influence Detection for Improving Efficiency in Reinforcement Learningcited by 71
  4. 2019Task-Relevant Adversarial Imitation Learningcited by 45
  5. 2023A Survey on Causal Reinforcement Learningcited by 45
  6. 2021Nonlinear Invariant Risk Minimization: A Causal Approachcited by 42
  7. 2021Causal Reinforcement Learning using Observational and Interventional Datacited by 39
  8. 2021Instrumental Variable Value Iteration for Causal Offline Reinforcement Learningcited by 35
  9. 2024The Essential Role of Causality in Foundation World Models for Embodied AIcited by 32
  10. 2021Reinforcement Learning for Education: Opportunities and Challengescited by 25
  11. 2021Sequential Deconfounding for Causal Inference with Unobserved Confounderscited by 24
  12. 2023Causal Reinforcement Learning: A Surveycited by 23
24 more
  1. 2023Explainable Reinforcement Learning via a Causal World Modelcited by 22
  2. 2021Dynamic Causal Bayesian Optimizationcited by 17
  3. 2022A Relational Intervention Approach for Unsupervised Dynamics Generalization in Model-Based Reinforcement Learningcited by 15
  4. 2024Causal Action Influence Aware Counterfactual Data Augmentationcited by 12
  5. 2024Fine-Grained Causal Dynamics Learning with Quantization for Improving Robustness in Reinforcement Learningcited by 11
  6. 2023Bad Habits: Policy Confounding and Out-of-Trajectory Generalization in RLcited by 10
  7. 2021Training a Resilient Q-Network against Observational Interferencecited by 9
  8. 2024Interpreting Low-level Vision Models with Causal Effect Mapscited by 8
  9. 2023Delphic Offline Reinforcement Learning under Nonidentifiable Hidden Confoundingcited by 7
  10. 2023Functional Causal Bayesian Optimizationcited by 5
  11. 2024Two-way Deconfounder for Off-policy Evaluation in Causal Reinforcement Learningcited by 5
  12. 2022Causal Deep Reinforcement Learning Using Observational Datacited by 3
  13. 2022Causal Temporal Reasoning for Markov Decision Processescited by 3
  14. 2024Learning Causal Dynamics Models in Object-Oriented Environmentscited by 3
  15. 2024Why Online Reinforcement Learning is Causalcited by 1
  16. 2026Causal Reinforcement Learning for Complex Card Games: A Magic The Gathering Benchmarkcited by 1
  17. 2021Adaptive Multi-Source Causal Inferencecited by 0
  18. 2022Statistical Estimation of Confounded Linear MDPs: An Instrumental Variable Approachcited by 0
  19. 2024Pessimistic Causal Reinforcement Learning with Mediators for Confounded Offline Datacited by 0
  20. 2025CAIMAN: Causal Action Influence Detection for Sample-efficient Loco-manipulationcited by 0
  21. 2025Parameter Estimation using Reinforcement Learning Causal Curiosity: Limits and Challengescited by 0
  22. 2025Beyond Prompt Engineering: Neuro-Symbolic-Causal Architecture for Robust Multi-Objective AI Agentscited by 0
  23. 2026Confounding Masquerading as Improvement: A Systematic Evaluation of Offline Reinforcement Learning for Stroke Antithrombotic Treatment in a 129,000-Patient Registrycited by 0
  24. 2026Temporal-Causal Inference for Reinforcement Learning via Automata Learningcited by 0