mlsanityData to 7 Oct 2026

Robust Fitted-Q-Evaluation and Iteration under Sequentially Exogenous Unobserved Confounders

arXiv 2302.00662 ↗First posted 1 Feb 2023stat.ML cs.LG math.OCcited by 12

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning is important in domains such as medicine, economics, and e-commerce where online experimentation is costly, dangerous or unethical, and where the true model is unknown. However, most methods assume all covariates used in the behavior policy's action decisions are observed. Though this assumption, sequential ignorability/unconfoundedness, likely does not hold in observational data, most of the data that accounts for selection into treatment may be observed, motivating sensitivity analysis. We study robust policy evaluation and policy optimization in the presence of sequentially-exogenous unobserved confounders under a sensitivity model. We propose and analyze orthogonalized robust fitted-Q-iteration that uses closed-form solutions of the robust Bellman operator to derive a loss minimization problem for the robust Q function, and adds a bias-correction to quantile estimation. Our algorithm enjoys the computational ease of fitted-Q-iteration and statistical improvements (reduced dependence on quantile estimation error) from orthogonalization. We provide sample complexity bounds, insights, and show effectiveness both in simulations and on real-world longitudinal healthcare data of treating sepsis. In particular, our model of sequential unobserved confounders yields an online Markov decision process, rather than partially observed Markov decision process: we illustrate how this can enable warm-starting optimistic reinforcement learning algorithms with valid robust bounds from observational data.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 60 of its 127 references that are in the map. First posted on arXiv 1 Feb 2023.

Builds on

60 of 127 references.

Closest ancestors · 9

Same direction, newest first

  1. 2022Offline Reinforcement Learning with Instrumental Variables in Confounded Markov Decision Processes18 Sep
  2. 2022Distributionally Robust Offline Reinforcement Learning with Linear Function Approximation14 Sep
  3. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
  4. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning9 Jun
  5. 2021Risk Bounds and Rademacher Complexity in Batch Reinforcement Learning25 Mar
  6. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism22 Mar
  7. 2021Instrumental Variable Value Iteration for Causal Offline Reinforcement Learning19 Feb
  8. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  9. 2020Confounding-Robust Policy Evaluation in Infinite-Horizon Reinforcement Learning11 Feb

Other references · 51

  1. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 514
  2. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
  3. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
  4. 2019Batch Policy Learning under Constraintscited by 242
  5. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 226
  6. 2015Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approachcited by 162
  7. 2018Learning Without Mixing: Towards A Sharp Analysis of Linear System Identificationcited by 158
  8. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 158
  9. 2019Orthogonal Statistical Learningcited by 150
  10. 2022Semiparametric doubly robust targeted double machine learning: a reviewcited by 129
  11. 2019Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processescited by 118
  12. 2014Generalization and Exploration via Randomized Value Functionscited by 112
38 more
  1. 2022Robust Reinforcement Learning using Offline Datacited by 95
  2. 2021Benchmarks for Deep Off-Policy Evaluationcited by 89
  3. 2017Deep Reinforcement Learning for Sepsis Treatmentcited by 71
  4. 2019Off-Policy Evaluation in Partially Observable Environmentscited by 65
  5. 2020Off-policy Policy Evaluation For Sequential Decisions Under Unobserved Confoundingcited by 60
  6. 2018Interval Estimation of Individual-Level Causal Effects Under Unobserved Confoundingcited by 51
  7. 2019Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimationcited by 48
  8. 2020Provably Efficient Causal Reinforcement Learning with Confounded Observational Datacited by 40
  9. 2020Off-policy Evaluation in Infinite-Horizon Reinforcement Learning with Latent Confounderscited by 40
  10. 2018An Off-policy Policy Gradient Theorem Using Emphatic Weightingscited by 34
  11. 2020An Empirical Study of Representation Learning for Reinforcement Learning in Healthcarecited by 34
  12. 2021A Minimax Learning Approach to Off-Policy Evaluation in Confounded Partially Observable Markov Decision Processescited by 34
  13. 2017Sensitivity analysis for inverse probability weighting estimators via the percentile bootstrapcited by 33
  14. 2021Sharp Sensitivity Analysis for Inverse Propensity Weighting via Quantile Balancingcited by 32
  15. 2022Off-Policy Confidence Interval Estimation with Confounded Markov Decision Processcited by 31
  16. 2021Doubly-Valid/Doubly-Sharp Sensitivity Analysis for Causal Inference with Unmeasured Confoundingcited by 28
  17. 2020Double/Debiased Machine Learning for Dynamic Treatment Effects via g-Estimationcited by 27
  18. 2020Statistically Efficient Off-Policy Policy Gradientscited by 24
  19. 2021Long Story Short: Omitted Variable Bias in Causal Machine Learningcited by 23
  20. 2020Bandits with Partially Observable Confounded Datacited by 22
  21. 2018Model-Based Reinforcement Learning for Sepsis Treatmentcited by 21
  22. 2022Scalable Sensitivity and Uncertainty Analysis for Causal-Effect Estimates of Continuous-Valued Interventionscited by 20
  23. 2022Future-Dependent Value-Based Off-Policy Evaluation in POMDPscited by 20
  24. 2022Off-Policy Evaluation for Episodic Partially Observable Markov Decision Processes under Non-Parametric Modelscited by 18
  25. 2019More Efficient Off-Policy Evaluation through Regularized Targeted Learningcited by 17
  26. 2019Sensitivity analysis via the proportion of unmeasured confoundingcited by 13
  27. 2021Estimating and Improving Dynamic Treatment Regimes With a Time-Varying Instrumental Variablecited by 13
  28. 2022Debiased Machine Learning without Sample-Splitting for Stable Estimatorscited by 13
  29. 2022Model-Free and Model-Based Policy Evaluation when Causality is Uncertaincited by 12
  30. 2022Sensitivity Analysis for Marginal Structural Modelscited by 10
  31. 2019Policy Evaluation with Latent Confounders via Optimal Balancecited by 8
  32. 2021Semiparametric sensitivity analysis: unmeasured confounding in observational studiescited by 8
  33. 2021Stateful Offline Contextual Policy Evaluation and Learningcited by 6
  34. 2021Ambiguous Dynamic Treatment Regimes: A Reinforcement Learning Approachcited by 6
  35. 2017Debiased Machine Learning of Set-Identified Linear Modelscited by 5
  36. 2020Assessing External Validity Over Worst-case Subpopulationscited by 5
  37. 2018Calculating CVaR and bPOE for Common Probability Distributions With Application to Portfolio Optimization and Density Estimationcited by 3
  38. 2021Uniformly Conservative Exploration in Reinforcement Learningcited by 3

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 6; 2025: 4; 2026: 1.

12 papers.

  1. 2025Offline Reinforcement Learning in Large State Spaces: Algorithms and Guaranteescited by 10
  2. 2023A Policy Gradient Method for Confounded POMDPscited by 6
  3. 2025Confounding Robust Deep Reinforcement Learning: A Causal Approachcited by 6
  4. 2024A Natural Extension To Online Algorithms For Hybrid RL With Limited Coveragecited by 5
  5. 2024Two-way Deconfounder for Off-policy Evaluation in Causal Reinforcement Learningcited by 5
  6. 2025Efficient and Sharp Off-Policy Learning under Unobserved Confoundingcited by 5
  7. 2024Efficient and Sharp Off-Policy Evaluation in Robust Markov Decision Processescited by 3
  8. 2024Model-Free Robust φ-Divergence Reinforcement Learning Using Both Offline and Online Datacited by 2
  9. 2024Causal Deepsets for Off-policy Evaluation under Spatial or Spatio-temporal Interferencescited by 2
  10. 2025Demystifying the Paradox of Importance Sampling with an Estimated History-Dependent Behavior Policy in Off-Policy Evaluationcited by 1
  11. 2024Pessimistic Causal Reinforcement Learning with Mediators for Confounded Offline Datacited by 0
  12. 2026Robust Counterfactual Policy Optimisation via Nondeterministic Causal Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Decision-Centered Abstractions via Orthogonal Estimation of Difference-of-Q Functions
  2. 2022Pessimism in the Face of Confounders: Provably Efficient Offline Reinforcement Learning in Partially Observable Markov Decision Processes
  3. 2022Offline Policy Evaluation and Optimization under Confounding
  4. 2023Delphic Offline Reinforcement Learning under Nonidentifiable Hidden Confounding
  5. 2026Learning from the Unseen: Offline Reinforcement Learning with Hidden Actions
  6. 2023Towards Robust Offline Reinforcement Learning under Diverse Data Corruption
  7. 2025Quantile Q-Learning: Revisiting Offline Extreme Q-Learning with Quantile Regression
  8. 2026Learning Optimal and Sample-Efficient Decision Policies with Guarantees