mlsanityData to 7 Oct 2026

Delphic Offline Reinforcement Learning under Nonidentifiable Hidden Confounding

arXiv 2306.01157 ↗First posted 1 Jun 2023cs.LG cs.AIcited by 7

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

A prominent challenge of offline reinforcement learning (RL) is the issue of hidden confounding: unobserved variables may influence both the actions taken by the agent and the observed outcomes. Hidden confounding can compromise the validity of any causal conclusion drawn from data and presents a major obstacle to effective offline RL. In the present paper, we tackle the problem of hidden confounding in the nonidentifiable setting. We propose a definition of uncertainty due to hidden confounding bias, termed delphic uncertainty, which uses variation over world models compatible with the observations, and differentiate it from the well-known epistemic and aleatoric uncertainties. We derive a practical method for estimating the three types of uncertainties, and construct a pessimistic offline RL algorithm to account for them. Our method does not assume identifiability of the unobserved confounders, and attempts to reduce the amount of confounding bias. We demonstrate through extensive experiments and ablations the efficacy of our approach on a sepsis management benchmark, as well as on electronic health records. Our results suggest that nonidentifiable hidden confounding bias can be mitigated to improve offline RL solutions in practice.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 62 of its 88 references that are in the map. First posted on arXiv 1 Jun 2023.

Builds on

62 of 88 references.

Closest ancestors · 16

Same direction, newest first

  1. 2023Leveraging Factored Action Spaces for Efficient Offline Reinforcement Learning in Healthcare2 May
  2. 2022Pessimism in the Face of Confounders: Provably Efficient Offline Reinforcement Learning in Partially Observable Markov Decision Processes26 May
  3. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learning5 Feb
  4. 2021d3rlpy: An Offline Deep Reinforcement Learning Library6 Nov
  5. 2021Proximal Reinforcement Learning: Efficient Off-Policy Evaluation in Partially Observed Markov Decision Processes28 Oct
  6. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage13 Jul
  7. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
  8. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  9. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  10. 2021Uncertainty Estimation Using Riemannian Model Dynamics for Offline Reinforcement Learning22 Feb
  11. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  12. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
4 more
  1. 2020MOPO: Model-based Offline Policy Optimization27 May
  2. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  4. 2020Confounding-Robust Policy Evaluation in Infinite-Horizon Reinforcement Learning11 Feb

Other references · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  3. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  4. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  5. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  7. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  8. 2019Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methodscited by 903
  9. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  10. 2016Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithmcited by 498
  11. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 457
  12. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
34 more
  1. 2019Causal Confusion in Imitation Learningcited by 286
  2. 2019Batch Policy Learning under Constraintscited by 242
  3. 2018Randomized Prior Functions for Deep Reinforcement Learningcited by 218
  4. 2015Contextual Markov Decision Processescited by 213
  5. 2021Epistemic Neural Networkscited by 120
  6. 2019Counterfactual Off-Policy Evaluation with Gumbel-Max Structural Causal Modelscited by 114
  7. 2018The Blessings of Multiple Causescited by 103
  8. 2020The Importance of Pessimism in Fixed-Dataset Policy Optimizationcited by 99
  9. 2018Confounding-Robust Policy Improvementcited by 73
  10. 2017Deep Reinforcement Learning for Sepsis Treatmentcited by 71
  11. 2021Shaking the foundations: delusions in sequence models for interaction and controlcited by 61
  12. 2018Evaluating Reinforcement Learning Algorithms in Observational Health Settingscited by 60
  13. 2020Off-policy Policy Evaluation For Sequential Decisions Under Unobserved Confoundingcited by 60
  14. 2020Identifying Causal-Effect Inference Failure with Uncertainty-Aware Modelscited by 57
  15. 2018Interval Estimation of Individual-Level Causal Effects Under Unobserved Confoundingcited by 51
  16. 2021Quantifying Ignorance in Individual-Level Causal-Effect Estimates under Hidden Confoundingcited by 50
  17. 2020Provably Efficient Causal Reinforcement Learning with Confounded Observational Datacited by 40
  18. 2020Off-policy Evaluation in Infinite-Horizon Reinforcement Learning with Latent Confounderscited by 40
  19. 2019What does it mean to solve the problem of discrimination in hiring? Social, technical and legal perspectives from the UK on automated hiring systemscited by 38
  20. 2021HiRID-ICU-Benchmark -- A Comprehensive Machine Learning Benchmark on High-resolution ICU Datacited by 38
  21. 2018Deconfounding Reinforcement Learning in Observational Settingscited by 36
  22. 2022Sequence Model Imitation Learning with Unobserved Contextscited by 34
  23. 2022Off-Policy Confidence Interval Estimation with Confounded Markov Decision Processcited by 31
  24. 2021A Critical Look at the Consistency of Causal Estimation With Deep Latent Variable Modelscited by 29
  25. 2023B-Learner: Quasi-Oracle Bounds on Heterogeneous Causal Effects Under Hidden Confoundingcited by 28
  26. 2016Contextual Bandits with Latent Confounders: An NMF Approachcited by 26
  27. 2021Reinforcement Learning for Education: Opportunities and Challengescited by 25
  28. 2022Sequential Causal Imitation Learning with Unobserved Confounderscited by 25
  29. 2020Bandits with Partially Observable Confounded Datacited by 22
  30. 2021On Stein Variational Neural Network Ensemblescited by 21
  31. 2021Invariant Policy Learning: A Causal Perspectivecited by 17
  32. 2021On Covariate Shift of Latent Confounders in Imitation and Reinforcement Learningcited by 14
  33. 2022POETREE: Interpretable Policy Learning with Adaptive Decision Treescited by 14
  34. 2023A Unified Framework of Policy Learning for Contextual Bandit with Confounding Bias and Missing Observationscited by 2

Built on by

Cited by year

Citing papers per year: 2024: 5; 2025: 2.

7 papers.

  1. 2024Learning Decision Policies with Instrumental Variables through Double Machine Learningcited by 5
  2. 2025Efficient and Sharp Off-Policy Learning under Unobserved Confoundingcited by 5
  3. 2024Preference Elicitation for Offline Reinforcement Learningcited by 3
  4. 2024BECAUSE: Bilinear Causal Representation for Generalizable Offline Model-based Reinforcement Learningcited by 3
  5. 2024Empowering Clinicians with Medical Decision Transformers: A Framework for Sepsis Treatmentcited by 2
  6. 2024Benchmarks for Reinforcement Learning with Biased Offline Data and Imperfect Simulatorscited by 0
  7. 2025Causal Policy Learning in Reinforcement Learning: Backdoor-Adjusted Soft Actor-Criticcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Offline Policy Evaluation and Optimization under Confounding
  2. 2022Offline Reinforcement Learning with Instrumental Variables in Confounded Markov Decision Processes
  3. 2026Learning from the Unseen: Offline Reinforcement Learning with Hidden Actions
  4. 2021False Correlation Reduction for Offline Reinforcement Learning
  5. 2022Offline Reinforcement Learning Under Value and Density-Ratio Realizability: The Power of Gaps
  6. 2021Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximation
  7. 2023Selective Uncertainty Propagation in Offline RL
  8. 2024Diffusion Policies for Risk-Averse Behavior Modeling in Offline Reinforcement Learning