# Evaluating the Surrogate Index as a Decision-Making Tool Using 200 A/B Tests at Netflix

arXiv: https://arxiv.org/abs/2311.11922 · HTML: https://mlsanity.com/paper/2311.11922/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Nov 2023.

## Built on by (most cited first, 6 of 6)

- [Predictions as Surrogates: Revisiting Surrogate Outcomes in the Age of AI](https://mlsanity.com/paper/2501.09731/) (2025)
- [Predicting Long-Term Student Outcomes from Short-Term EdTech Log Data](https://mlsanity.com/paper/2412.15473/) (2024)
- [Learning plug-in surrogate endpoints for randomized experiments](https://mlsanity.com/paper/2605.12051/) (2026)
- [Short-Long Policy Evaluation with Novel Actions](https://mlsanity.com/paper/2407.03674/) (2024)
- [Predicting Long Term Sequential Policy Value Using Softer Surrogates](https://mlsanity.com/paper/2412.20638/) (2024)
- [From Offline Proxies to Online Decisions: A Layered Engagement Evaluation Framework for Conversational AI](https://mlsanity.com/paper/2609.25408/) (2026)

## Abstract (arXiv)

Surrogate index approaches have recently become a popular method of estimating longer-term impact from shorter-term outcomes. In this paper, we leverage 1098 test arms from 200 A/B tests at Netflix to empirically investigate to what degree would decisions made using a surrogate index utilizing 14 days of data would align with those made using direct measurement of day 63 treatment effects. Focusing specifically on linear "auto-surrogate" models that utilize the shorter-term observations of the long-term outcome of interest, we find that the statistical inferences that we would draw from using the surrogate index are ~95% consistent with those from directly measuring the long-term treatment effect. Moreover, when we restrict ourselves to the set of tests that would be "launched" (i.e. positive and statistically significant) based on the 63-day directly measured treatment effects, we find that relying instead on the surrogate index achieves 79% and 65% recall.