# Choosing a Proxy Metric from Past Experiments

arXiv: https://arxiv.org/abs/2309.07893 · HTML: https://mlsanity.com/paper/2309.07893/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 33 references that are in the map.
- First posted on arXiv 14 Sep 2023.

## Builds on (most cited first, 4 of 4)

- [Composable Effects for Flexible and Accelerated Probabilistic Programming in NumPyro](https://mlsanity.com/paper/1912.11554/) (2019)
- [On the role of surrogates in the efficient estimation of treatment effects with limited outcome data](https://mlsanity.com/paper/2003.12408/) (2020)
- [Pareto optimal proxy metrics](https://mlsanity.com/paper/2307.01000/) (2023)
- [Evaluating Surrogate Marker Information using Censored Data](https://mlsanity.com/paper/1602.05894/) (2016)

## Built on by (most cited first, 11 of 11)

- [Learning Metrics that Maximise Power for Accelerated A/B-Tests](https://mlsanity.com/paper/2402.03915/) (2024)
- [Reinforcement Learning in the Real World: A Survey of Statistical Challenges and Future Directions](https://mlsanity.com/paper/2601.15353/) (2026)
- [t-Testing the Waters: Empirically Validating Assumptions for Reliable A/B-Testing](https://mlsanity.com/paper/2502.04793/) (2025)
- [Reduced-Rank Multi-objective Policy Learning and Optimization](https://mlsanity.com/paper/2404.18490/) (2024)
- [Estimate Level Adjustment For Inference With Proxies Under Random Distribution Shifts](https://mlsanity.com/paper/2605.06484/) (2026)
- [Learning plug-in surrogate endpoints for randomized experiments](https://mlsanity.com/paper/2605.12051/) (2026)
- [PROXIMA: A Reliability Scoring Framework for Proxy Metrics in Online Controlled Experiments](https://mlsanity.com/paper/2604.14352/) (2026)
- [Accelerating A/B-Tests with Counterfactual Estimation: Reducing Variance through Policy Overlap](https://mlsanity.com/paper/2607.14604/) (2026)
- [proxymate: Diagnosis and Adjustment of Proxy Estimates for Reliable Inference](https://mlsanity.com/paper/2607.24401/) (2026)
- [When Offline Evaluation Misleads: A Diagnostic Protocol for Reward and Policy Selection in Delayed-Feedback Contextual Bandits](https://mlsanity.com/paper/2608.11560/) (2026)
- [From Offline Proxies to Online Decisions: A Layered Engagement Evaluation Framework for Conversational AI](https://mlsanity.com/paper/2609.25408/) (2026)

## Abstract (arXiv)

In many randomized experiments, the treatment effect of the long-term metric (i.e. the primary outcome of interest) is often difficult or infeasible to measure. Such long-term metrics are often slow to react to changes and sufficiently noisy they are challenging to faithfully estimate in short-horizon experiments. A common alternative is to measure several short-term proxy metrics in the hope they closely track the long-term metric -- so they can be used to effectively guide decision-making in the near-term. We introduce a new statistical framework to both define and construct an optimal proxy metric for use in a homogeneous population of randomized experiments. Our procedure first reduces the construction of an optimal proxy metric in a given experiment to a portfolio optimization problem which depends on the true latent treatment effects and noise level of experiment under consideration. We then denoise the observed treatment effects of the long-term metric and a set of proxies in a historical corpus of randomized experiments to extract estimates of the latent treatment effects for use in the optimization problem. One key insight derived from our approach is that the optimal proxy metric for a given experiment is not apriori fixed; rather it should depend on the sample size (or effective noise level) of the randomized experiment for which it is deployed. To instantiate and evaluate our framework, we employ our methodology in a large corpus of randomized experiments from an industrial recommendation system and construct proxy metrics that perform favorably relative to several baselines.