mlsanityData to 14 Oct 2026

A Unified Framework of Policy Learning for Contextual Bandit with Confounding Bias and Missing Observations

arXiv 2303.11187 ↗First posted 20 Mar 2023cs.LG cs.AI stat.MLcited by 2

Not on the map. Why?

Abstract (arXiv)

We study the offline contextual bandit problem, where we aim to acquire an optimal policy using observational data. However, this data usually contains two deficiencies: (i) some variables that confound actions are not observed, and (ii) missing observations exist in the collected data. Unobserved confounders lead to a confounding bias and missing observations cause bias and inefficiency problems. To overcome these challenges and learn the optimal policy from the observed dataset, we present a new algorithm called Causal-Adjusted Pessimistic (CAP) policy learning, which forms the reward function as the solution of an integral equation system, builds a confidence set, and greedily takes action with pessimism. With mild assumptions on the data, we develop an upper bound to the suboptimality of CAP for the offline contextual bandit problem.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 45 of its 90 references that are in the map. First posted on arXiv 20 Mar 2023.

Builds on

45 of 90 references.

References · 45

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
  3. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  5. 2017Minimax Regret Bounds for Reinforcement Learningcited by 503
  6. 2020Is Pessimism Provably Efficient for Offline RL?cited by 364
  7. 2016Safe, Multi-Agent, Reinforcement Learning for Autonomous Drivingcited by 311
  8. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 287
  9. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 279
  10. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 269
  11. 2015Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hourscited by 209
  12. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 144
33 more
  1. 2016Identifying Causal Effects With Proxy Variables of an Unmeasured Confoundercited by 139
  2. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrabilitycited by 123
  3. 2020Minimax-Optimal Off-Policy Evaluation with Linear Function Approximationcited by 120
  4. 2019Kernel Instrumental Variable Regressioncited by 118
  5. 2020The Importance of Pessimism in Fixed-Dataset Policy Optimizationcited by 99
  6. 2020An Introduction to Proximal Causal Learningcited by 95
  7. 2019Deep Generalized Method of Moments for Instrumental Variable Analysiscited by 88
  8. 2020Off-Policy Evaluation via the Regularized Lagrangiancited by 86
  9. 2020Minimax Estimation of Conditional Moment Modelscited by 83
  10. 2021Towards Instance-Optimal Offline Reinforcement Learning with Pessimismcited by 78
  11. 2020Provably Good Batch Reinforcement Learning Without Great Explorationcited by 77
  12. 2020Batch Policy Learning in Average Reward Markov Decision Processescited by 69
  13. 2022Near-optimal Offline Reinforcement Learning with Linear Representation: Leveraging Variance Information with Pessimismcited by 65
  14. 2020Semiparametric proximal causal inferencecited by 59
  15. 2019Estimating Treatment Effects with Causal Forests: An Applicationcited by 58
  16. 2021Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiencycited by 56
  17. 2020Exponential Lower Bounds for Batch Reinforcement Learning: Batch RL can be Exponentially Harder than Online RLcited by 55
  18. 2017A Reinforcement Learning Approach to Weaning of Mechanical Ventilation in Intensive Care Unitscited by 50
  19. 2021Risk Bounds and Rademacher Complexity in Batch Reinforcement Learningcited by 42
  20. 2022The Efficacy of Pessimism in Asynchronous Q-Learningcited by 42
  21. 2021Proximal Reinforcement Learning: Efficient Off-Policy Evaluation in Partially Observed Markov Decision Processescited by 41
  22. 2018A confounding bridge approach for double negative control inference on causal effectscited by 36
  23. 2021Instrumental Variable Value Iteration for Causal Offline Reinforcement Learningcited by 35
  24. 2020Kernel Methods for Unobserved Confounding: Negative Controls, Proxies, and Instrumentscited by 28
  25. 2020Bandits with Partially Observable Confounded Datacited by 22
  26. 2021Proximal Learning for Individualized Treatment Regimes Under Unmeasured Confoundingcited by 21
  27. 2021A Spectral Approach to Off-Policy Evaluation for POMDPscited by 21
  28. 2018Semiparametric Contextual Banditscited by 19
  29. 2021Proximal Causal Inference for Complex Longitudinal Studiescited by 16
  30. 2021Estimating and Improving Dynamic Treatment Regimes With a Time-Varying Instrumental Variablecited by 13
  31. 2023Minimax Instrumental Variable Regression and L2 Convergence Guarantees without Identification or Closednesscited by 13
  32. 2017Causal inference with confounders missing not at randomcited by 7
  33. 2021Efficient Estimation in NPIV Models: A Comparison of Various Neural Networks-Based Estimatorscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 0; 2025: 0; 2026: 1.

2 papers.

  1. 2023Delphic Offline Reinforcement Learning under Nonidentifiable Hidden Confoundingcited by 7
  2. 2026What Do We Care About in Bandits with Noncompliance? BRACE: Bandits with Recommendations, Abstention, and Certified Effectscited by 0