# Optimizing Returns from Experimentation Programs

arXiv: https://arxiv.org/abs/2412.05508 · HTML: https://mlsanity.com/paper/2412.05508/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Dec 2024.

## Built on by (most cited first, 2 of 2)

- [Incentive Alignment in Online Experimentation](https://mlsanity.com/paper/2610.05922/) (2026)
- [Policy Learning with Weak Signals](https://mlsanity.com/paper/2610.10167/) (2026)

## Abstract (arXiv)

Experimentation in online digital platforms is used to inform decision making. Specifically, the goal of many experiments is to optimize a metric of interest. Null hypothesis statistical testing can be ill-suited to this task, as it is indifferent to the magnitude of effect sizes and opportunity costs. Given access to a pool of related past experiments, we discuss how experimentation practice should change when the goal is optimization. We survey the literature on empirical Bayes analyses of A/B test portfolios, and single out the A/B Testing Problem (Azevedo et al., 2020) as a starting point, which treats experimentation as a constrained optimization problem. We show that the framework can be solved with dynamic programming and implemented by appropriately tuning p-value thresholds. Furthermore, we develop several extensions of the A/B Testing Problem and discuss the implications of these results on experimentation programs in industry. For example, under no-cost assumptions, firms should be testing many more ideas, reducing test allocation sizes, and relaxing p-value thresholds away from p = 0.05.