mlsanityData to 7 Oct 2026

Confounding Robust Deep Reinforcement Learning: A Causal Approach

arXiv 2510.21110 ↗First posted 24 Oct 2025cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

A key task in Artificial Intelligence is learning effective policies for controlling agents in unknown environments to optimize performance measures. Off-policy learning methods, like Q-learning, allow learners to make optimal decisions based on past experiences. This paper studies off-policy learning from biased data in complex and high-dimensional domains where unobserved confounding cannot be ruled out a priori. Building on the well-celebrated Deep Q-Network (DQN), we propose a novel deep reinforcement learning algorithm robust to confounding biases in observed data. Specifically, our algorithm attempts to find a safe policy for the worst-case environment compatible with the observations. We apply our method to twelve confounded Atari games, and find that it consistently dominates the standard DQN in all games where the observed input to the behavioral and target policies mismatch and unobserved confounders exist.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 57 of its 117 references that are in the map. First posted on arXiv 24 Oct 2025.

Builds on

57 of 117 references.

References · 57

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  4. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  5. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  6. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  7. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  8. 2017Hindsight Experience Replaycited by 1,096
  9. 2015Prioritized Experience Replaycited by 1,070
  10. 2018Exploration by Random Network Distillationcited by 966
  11. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  12. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
38 more
  1. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  2. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  3. 2014Deep Learning in Neural Networks: An Overviewcited by 765
  4. 2019Solving Rubik's Cube with a Robot Handcited by 699
  5. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 698
  6. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 457
  7. 2017Robust Adversarial Reinforcement Learningcited by 439
  8. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
  9. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  10. 2020Agent57: Outperforming the Atari Human Benchmarkcited by 300
  11. 2024Diffusion for World Modeling: Visual Details Matter in Ataricited by 285
  12. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 260
  13. 2020Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observationscited by 235
  14. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
  15. 2017DeepStack: Expert-Level Artificial Intelligence in No-Limit Pokercited by 228
  16. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 158
  17. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 145
  18. 2017Visualizing and Understanding Atari Agentscited by 141
  19. 2018Reinforcement and Imitation Learning for Diverse Visuomotor Skillscited by 124
  20. 2021Online Robust Reinforcement Learning with Model Uncertaintycited by 95
  21. 2022Robust Reinforcement Learning using Offline Datacited by 95
  22. 2020An Introduction to Proximal Causal Learningcited by 91
  23. 2019Observational Overfitting in Reinforcement Learningcited by 90
  24. 2013Probabilities of Causation: Bounds and Identificationcited by 89
  25. 2023RoboCook: Long-Horizon Elasto-Plastic Object Manipulation with Diverse Toolscited by 75
  26. 2018Confounding-Robust Policy Improvementcited by 73
  27. 2021Partial Counterfactual Identification from Observational and Experimental Datacited by 64
  28. 2020Off-policy Policy Evaluation For Sequential Decisions Under Unobserved Confoundingcited by 60
  29. 2020Confounding-Robust Policy Evaluation in Infinite-Horizon Reinforcement Learningcited by 56
  30. 2017Reinforcement Learning under Model Mismatchcited by 52
  31. 2016Safe Policy Improvement by Minimizing Robust Baseline Regretcited by 51
  32. 2020Off-policy Evaluation in Infinite-Horizon Reinforcement Learning with Latent Confounderscited by 40
  33. 2021Proximal Reinforcement Learning: Efficient Off-Policy Evaluation in Partially Observed Markov Decision Processescited by 40
  34. 2022Causal Imitation Learning with Unobserved Confounderscited by 40
  35. 2019Beyond Confidence Regions: Tight Bayesian Ambiguity Sets for Robust MDPscited by 37
  36. 2021Podracer architectures for scalable Reinforcement Learningcited by 37
  37. 2021A Minimax Learning Approach to Off-Policy Evaluation in Confounded Partially Observable Markov Decision Processescited by 34
  38. 2015Distributionally Robust Counterpart in Markov Decision Processescited by 32

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Discovering What You Can Control: Interventional Boundary Discovery for Reinforcement Learningcited by 2
  2. 2026Foundation Models for Partial Causal Identificationcited by 2
  3. 2026Causal Flow Q-Learning for Robust Offline Reinforcement Learningcited by 1
  4. 2026Integrating Causal DAGs in Deep RL: Activating Minimal Markovian States with Multi-Order Exposurecited by 0
  5. 2026Scalable Causal Imitation Learningcited by 0
  6. 2026Compiling VGDL into Causal Modelscited by 0