mlsanityData to 7 Oct 2026

Towards Robust Off-Policy Evaluation via Human Inputs

arXiv 2209.08682 ↗First posted 18 Sep 2022cs.LG stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

Off-policy Evaluation (OPE) methods are crucial tools for evaluating policies in high-stakes domains such as healthcare, where direct deployment is often infeasible, unethical, or expensive. When deployment environments are expected to undergo changes (that is, dataset shifts), it is important for OPE methods to perform robust evaluation of the policies amidst such changes. Existing approaches consider robustness against a large class of shifts that can arbitrarily change any observable property of the environment. This often results in highly pessimistic estimates of the utilities, thereby invalidating policies that might have been useful in deployment. In this work, we address the aforementioned problem by investigating how domain knowledge can help provide more realistic estimates of the utilities of policies. We leverage human inputs on which aspects of the environments may plausibly change, and adapt the OPE methods to only consider shifts on these aspects. Specifically, we propose a novel framework, Robust OPE (ROPE), which considers shifts on a subset of covariates in the data based on user inputs, and estimates worst-case utility under these shifts. We then develop computationally efficient algorithms for OPE that are robust to the aforementioned shifts for contextual bandits and Markov decision processes. We also theoretically analyze the sample complexity of these algorithms. Extensive experimentation with synthetic and real world datasets from the healthcare domain demonstrates that our approach not only captures realistic dataset shifts accurately, but also results in less pessimistic policy evaluations.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 38 of its 68 references that are in the map. First posted on arXiv 18 Sep 2022.

Builds on

38 of 68 references.

References · 38

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  5. 2015Causal inference using invariant prediction: identification and confidence intervalscited by 533
  6. 2020Measuring Robustness to Natural Distribution Shifts in Image Classificationcited by 452
  7. 2021Retiring Adult: New Datasets for Fair Machine Learningcited by 443
  8. 2017Robust Adversarial Reinforcement Learningcited by 439
  9. 2018Learning Models with Uniform Performance via Distributionally Robust Optimizationcited by 334
  10. 2017Certifying Some Distributional Robustness with Principled Adversarial Trainingcited by 333
  11. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 285
  12. 2015Doubly Robust Policy Evaluation and Optimizationcited by 158
26 more
  1. 2020Understanding and Mitigating the Tradeoff Between Robustness and Accuracycited by 148
  2. 2020Perceptual Adversarial Robustness: Defense Against Unseen Threat Modelscited by 137
  3. 2017Domain Adaptation by Using Causal Inference to Predict Invariant Conditional Distributionscited by 125
  4. 2018Anchor regression: heterogeneous data meets causalitycited by 116
  5. 2019Counterfactual Off-Policy Evaluation with Gumbel-Max Structural Causal Modelscited by 114
  6. 2020Breaking the Sample Size Barrier in Model-Based Reinforcement Learning with a Generative Modelcited by 108
  7. 2020Invariant Causal Prediction for Block MDPscited by 104
  8. 2020The Effect of Natural Distribution Shift on Question Answering Modelscited by 104
  9. 2019Adversarial Robustness Against the Union of Multiple Perturbation Modelscited by 88
  10. 2013Data-Driven Robust Optimizationcited by 84
  11. 2020A causal framework for distribution generalizationcited by 68
  12. 2018Evaluating Reinforcement Learning Algorithms in Observational Health Settingscited by 60
  13. 2020Off-policy Policy Evaluation For Sequential Decisions Under Unobserved Confoundingcited by 60
  14. 2016Does Distributionally Robust Supervised Learning Give Robust Classifiers?cited by 38
  15. 2019Beyond Confidence Regions: Tight Bayesian Ambiguity Sets for Robust MDPscited by 37
  16. 2020POPCORN: Partially Observed Prediction COnstrained ReiNforcement Learningcited by 36
  17. 2020Learning Optimal Distributionally Robust Individualized Treatment Rulescited by 28
  18. 2019Distributionally Robust Counterfactual Risk Minimizationcited by 26
  19. 2020Off-Policy Evaluation and Learning for External Validity under a Covariate Shiftcited by 26
  20. 2021Feature Shift Detection: Localizing Which Features Have Shifted via Conditional Distribution Testscited by 24
  21. 2020Distributionally Robust Batch Contextual Banditscited by 20
  22. 2021Regularizing towards Causal Invariance: Linear Models with Proxiescited by 20
  23. 2021Generalizing Off-Policy Learning under Sample Selection Biascited by 16
  24. 2020Counterfactually Guided Off-policy Transfer in Clinical Settingscited by 13
  25. 2018Generalizability of predictive models for intensive care unit patientscited by 9
  26. 2020Robust Batch Policy Learning in Markov Decision Processescited by 4

Built on by

Not cited yet.