# Designing and Deploying Online Field Experiments

arXiv: https://arxiv.org/abs/1409.3174 · HTML: https://mlsanity.com/paper/1409.3174/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Sep 2014.

## Built on by (most cited first, 18 of 18)

- [Looper: An end-to-end ML platform for product decisions](https://mlsanity.com/paper/2110.07554/) (2021)
- [Policy Evaluation for Temporal and/or Spatial Dependent Experiments](https://mlsanity.com/paper/2202.10887/) (2022)
- [Adaptive Experimentation When You Can't Experiment](https://mlsanity.com/paper/2406.10738/) (2024)
- [Prediction-Guided Active Experiments](https://mlsanity.com/paper/2411.12036/) (2024)
- [On Efficient Estimation of Distributional Treatment Effects under Covariate-Adaptive Randomization](https://mlsanity.com/paper/2506.05945/) (2025)
- [Tisane: Authoring Statistical Models via Formal Reasoning from Conceptual and Data Relationships](https://mlsanity.com/paper/2201.02705/) (2022)
- [Estimating Distributional Treatment Effects in Randomized Experiments: Machine Learning for Variance Reduction](https://mlsanity.com/paper/2407.16037/) (2024)
- [A Bayesian Model for Online Activity Sample Sizes](https://mlsanity.com/paper/2111.12157/) (2021)
- [A Two-Part Machine Learning Approach to Characterizing Network Interference in A/B Testing](https://mlsanity.com/paper/2308.09790/) (2023)
- [Experimenting, Fast and Slow: Bayesian Optimization of Long-term Outcomes with Online Experiments](https://mlsanity.com/paper/2506.18744/) (2025)
- [Distilled Thompson Sampling: Practical and Efficient Thompson Sampling via Imitation Learning](https://mlsanity.com/paper/2011.14266/) (2020)
- [Algorithms for Adaptive Experiments that Trade-off Statistical Analysis with Reward: Combining Uniform Random Assignment and Reward Maximization](https://mlsanity.com/paper/2112.08507/) (2021)
- [Assign Experiment Variants at Scale in Online Controlled Experiments](https://mlsanity.com/paper/2212.08771/) (2022)
- [Practical Policy Optimization with Personalized Experimentation](https://mlsanity.com/paper/2303.17648/) (2023)
- [Balancing Risk and Reward: An Automated Phased Release Strategy](https://mlsanity.com/paper/2305.09626/) (2023)
- [Generalized Random Forests using Fixed-Point Trees](https://mlsanity.com/paper/2306.11908/) (2023)
- [A Nonparametric Bayes Approach to Online Activity Prediction](https://mlsanity.com/paper/2401.14722/) (2024)
- [Incentive Alignment in Online Experimentation](https://mlsanity.com/paper/2610.05922/) (2026)

## Abstract (arXiv)

Online experiments are widely used to compare specific design alternatives, but they can also be used to produce generalizable knowledge and inform strategic decision making. Doing so often requires sophisticated experimental designs, iterative refinement, and careful logging and analysis. Few tools exist that support these needs. We thus introduce a language for online field experiments called PlanOut. PlanOut separates experimental design from application code, allowing the experimenter to concisely describe experimental designs, whether common "A/B tests" and factorial designs, or more complex designs involving conditional logic or multiple experimental units. These latter designs are often useful for understanding causal mechanisms involved in user behaviors. We demonstrate how experiments from the literature can be implemented in PlanOut, and describe two large field experiments conducted on Facebook with PlanOut. For common scenarios in which experiments are run iteratively and in parallel, we introduce a namespaced management system that encourages sound experimental practice.