mlsanityData to 14 Oct 2026

Predicting Long Term Sequential Policy Value Using Softer Surrogates

arXiv 2412.20638 ↗First posted 30 Dec 2024cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Off-policy policy evaluation (OPE) estimates the outcome of a new policy using historical data collected from a different policy. However, existing OPE methods cannot handle cases when the new policy introduces novel actions. This issue commonly occurs in real-world domains, like healthcare, as new drugs and treatments are continuously developed. Novel actions necessitate on-policy data collection, which can be burdensome and expensive if the outcome of interest takes a substantial amount of time to observe--for example, in multi-year clinical trials. This raises a key question of how to predict the long-term outcome of a policy after only observing its short-term effects? Though in general this problem is intractable, under some surrogacy conditions, the short-term on-policy data can be combined with the long-term historical data to make accurate predictions about the new policy's long-term value. In two simulated healthcare examples--HIV and sepsis management--we show that our estimators can provide accurate predictions about the policy value only after observing 10% of the full horizon data. We also provide finite sample analysis of our doubly robust estimators.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 20 of its 37 references that are in the map. First posted on arXiv 30 Dec 2024.

Builds on

20 of 37 references.

References · 20

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  2. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 286
  3. 2019Counterfactual Off-Policy Evaluation with Gumbel-Max Structural Causal Modelscited by 117
  4. 2020Off-policy Policy Evaluation For Sequential Decisions Under Unobserved Confoundingcited by 61
  5. 2020On the role of surrogates in the efficient estimation of treatment effects with limited outcome datacited by 38
  6. 2018Best arm identification in multi-armed bandits with delayed feedbackcited by 29
  7. 2020Dynamic Causal Effects Evaluation in A/B Testing with a Reinforcement Learning Frameworkcited by 29
  8. 2023Impatient Bandits: Optimizing Recommendations for the Long-Term Without Delaycited by 17
  9. 2023Automatic Debiased Machine Learning for Covariate Shiftscited by 16
  10. 2020Long-Term Effect Estimation with Surrogate Representationcited by 15
  11. 2016Estimating Treatment Effects using Multiple Surrogates: The Role of the Surrogate Score and the Surrogate Indexcited by 12
  12. 2021A Simple and General Debiased Machine Learning Theorem with Finite Sample Guaranteescited by 9
8 more
  1. 2023Inferring the Long-Term Causal Effects of Long-Term Treatments from Short-Term Experimentscited by 9
  2. 2024Long-term Off-Policy Evaluation and Learningcited by 9
  3. 2021Estimating the Long-Term Effects of Novel Treatmentscited by 8
  4. 2023Improving Student Learning with Hybrid Human-AI Tutoring: A Three-Study Quasi-Experimental Investigationcited by 8
  5. 2022Offline Policy Optimization with Eligible Actionscited by 7
  6. 2023Evaluating the Surrogate Index as a Decision-Making Tool Using 200 A/B Tests at Netflixcited by 6
  7. 2022A Reinforcement Learning Approach to Estimating Long-term Treatment Effectscited by 5
  8. 2024Aligning Tutor Discourse Supporting Rigorous Thinking with Tutee Content Mastery for Predicting Math Achievementcited by 3

Built on by

Not cited yet.