# Predicting Long Term Sequential Policy Value Using Softer Surrogates

arXiv: https://arxiv.org/abs/2412.20638 · HTML: https://mlsanity.com/paper/2412.20638/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 37 references that are in the map.
- First posted on arXiv 30 Dec 2024.

## Builds on (most cited first, 20 of 20)

- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning](https://mlsanity.com/paper/1604.00923/) (2016)
- [Counterfactual Off-Policy Evaluation with Gumbel-Max Structural Causal Models](https://mlsanity.com/paper/1905.05824/) (2019)
- [Off-policy Policy Evaluation For Sequential Decisions Under Unobserved Confounding](https://mlsanity.com/paper/2003.05623/) (2020)
- [On the role of surrogates in the efficient estimation of treatment effects with limited outcome data](https://mlsanity.com/paper/2003.12408/) (2020)
- [Best arm identification in multi-armed bandits with delayed feedback](https://mlsanity.com/paper/1803.10937/) (2018)
- [Dynamic Causal Effects Evaluation in A/B Testing with a Reinforcement Learning Framework](https://mlsanity.com/paper/2002.01711/) (2020)
- [Impatient Bandits: Optimizing Recommendations for the Long-Term Without Delay](https://mlsanity.com/paper/2307.09943/) (2023)
- [Automatic Debiased Machine Learning for Covariate Shifts](https://mlsanity.com/paper/2307.04527/) (2023)
- [Long-Term Effect Estimation with Surrogate Representation](https://mlsanity.com/paper/2008.08236/) (2020)
- [Estimating Treatment Effects using Multiple Surrogates: The Role of the Surrogate Score and the Surrogate Index](https://mlsanity.com/paper/1603.09326/) (2016)
- [A Simple and General Debiased Machine Learning Theorem with Finite Sample Guarantees](https://mlsanity.com/paper/2105.15197/) (2021)
- [Inferring the Long-Term Causal Effects of Long-Term Treatments from Short-Term Experiments](https://mlsanity.com/paper/2311.08527/) (2023)
- [Long-term Off-Policy Evaluation and Learning](https://mlsanity.com/paper/2404.15691/) (2024)
- [Estimating the Long-Term Effects of Novel Treatments](https://mlsanity.com/paper/2103.08390/) (2021)
- [Improving Student Learning with Hybrid Human-AI Tutoring: A Three-Study Quasi-Experimental Investigation](https://mlsanity.com/paper/2312.11274/) (2023)
- [Offline Policy Optimization with Eligible Actions](https://mlsanity.com/paper/2207.00632/) (2022)
- [Evaluating the Surrogate Index as a Decision-Making Tool Using 200 A/B Tests at Netflix](https://mlsanity.com/paper/2311.11922/) (2023)
- [A Reinforcement Learning Approach to Estimating Long-term Treatment Effects](https://mlsanity.com/paper/2210.07536/) (2022)
- [Aligning Tutor Discourse Supporting Rigorous Thinking with Tutee Content Mastery for Predicting Math Achievement](https://mlsanity.com/paper/2405.06218/) (2024)

## Abstract (arXiv)

Off-policy policy evaluation (OPE) estimates the outcome of a new policy using historical data collected from a different policy. However, existing OPE methods cannot handle cases when the new policy introduces novel actions. This issue commonly occurs in real-world domains, like healthcare, as new drugs and treatments are continuously developed. Novel actions necessitate on-policy data collection, which can be burdensome and expensive if the outcome of interest takes a substantial amount of time to observe--for example, in multi-year clinical trials. This raises a key question of how to predict the long-term outcome of a policy after only observing its short-term effects? Though in general this problem is intractable, under some surrogacy conditions, the short-term on-policy data can be combined with the long-term historical data to make accurate predictions about the new policy's long-term value. In two simulated healthcare examples--HIV and sepsis management--we show that our estimators can provide accurate predictions about the policy value only after observing 10% of the full horizon data. We also provide finite sample analysis of our doubly robust estimators.