# Choosing Online Experiment Designs under Interference in Ads, Recommendations, and Member-Experience Systems

arXiv: https://arxiv.org/abs/2605.25290 · HTML: https://mlsanity.com/paper/2605.25290/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 37 references that are in the map.
- First posted on arXiv 24 May 2026.

## Builds on (most cited first, 23 of 23)

- [Open Bandit Dataset and Pipeline: Towards Realistic and Reproducible Off-Policy Evaluation](https://mlsanity.com/paper/2008.07146/) (2020)
- [KuaiRand: An Unbiased Sequential Recommendation Dataset with Randomly Exposed Videos](https://mlsanity.com/paper/2208.08696/) (2022)
- [Graph cluster randomization: network exposure to multiple universes](https://mlsanity.com/paper/1305.6979/) (2013)
- [Design and analysis of experiments in networks: Reducing bias from interference](https://mlsanity.com/paper/1404.7530/) (2014)
- [Experimental Design in Two-Sided Platforms: An Analysis of Bias](https://mlsanity.com/paper/2002.05670/) (2020)
- [Design and Analysis of Switchback Experiments](https://mlsanity.com/paper/2009.00148/) (2020)
- [Interference, Bias, and Variance in Two-Sided Marketplace Experimentation: Guidance for Platforms](https://mlsanity.com/paper/2104.12222/) (2021)
- [Estimating Total Treatment Effect in Randomized Experiments with Unknown Network Structure](https://mlsanity.com/paper/2205.12803/) (2022)
- [Randomized Graph Cluster Randomization](https://mlsanity.com/paper/2009.02297/) (2020)
- [Reducing Interference Bias in Online Marketplace Pricing Experiments](https://mlsanity.com/paper/2004.12489/) (2020)
- [A/B Testing for Recommender Systems in a Two-sided Marketplace](https://mlsanity.com/paper/2106.00762/) (2021)
- [Causal clustering: design of cluster experiments under network interference](https://mlsanity.com/paper/2310.14983/) (2023)
- [Limiting Bias from Test-Control Interference in Online Marketplace Experiments](https://mlsanity.com/paper/2004.12162/) (2020)
- [Limitations of design-based causal inference and A/B testing under arbitrary and network interference](https://mlsanity.com/paper/1705.05752/) (2017)
- [Causal Inference under Network Interference Using a Mixture of Randomized Experiments](https://mlsanity.com/paper/2309.00141/) (2023)
- [Statistical Inference and A/B Testing for First-Price Pacing Equilibria](https://mlsanity.com/paper/2301.02276/) (2023)
- [Producer-Side Experiments Based on Counterfactual Interleaving Designs for Online Recommender Systems](https://mlsanity.com/paper/2310.16294/) (2023)
- [Interference Among First-Price Pacing Equilibria: A Bias and Variance Analysis](https://mlsanity.com/paper/2402.07322/) (2024)
- [Seller-Side Experiments under Interference Induced by Feedback Loops in Two-Sided Platforms](https://mlsanity.com/paper/2401.15811/) (2024)
- [Cascade-based Randomization for Inferring Causal Effects under Diffusion Interference](https://mlsanity.com/paper/2405.12340/) (2024)
- [Robust and efficient multiple-unit switchback experimentation](https://mlsanity.com/paper/2506.12654/) (2025)
- [Decision Support for Marketplace Policies under Incomplete Evidence: From Replay to Launch Readiness](https://mlsanity.com/paper/2605.12840/) (2026)
- [Assign Experiment Variants at Scale in Online Controlled Experiments](https://mlsanity.com/paper/2212.08771/) (2022)

## Built on by (most cited first, 1 of 1)

- [Privacy-Robust Incrementality Measurement for Advertising Systems under Signal Loss](https://mlsanity.com/paper/2606.03878/) (2026)

## Abstract (arXiv)

Online experiments in ads, recommendation, and member-experience systems are often planned before the dominant interference mechanism is known. A treatment may propagate through budgets, inventory, producer exposure, graph spillovers, or temporal carryover, making the randomization design itself a statistical decision. We formulate this problem as robust design selection over uncertain exposure mechanisms. Given a finite catalog of six implementable designs, the selector compares each design by worst-case planning risk over an ambiguity set. The risk combines exposure bias, assignment-unit variance, minimum detectable effect, contamination or carryover, operational cost, and estimand mismatch. For theoretical justification, the paper develops a geometry-aware guarantee, stating that design bias is bounded by Wasserstein distance to the launch exposure distribution, and this penalty is minimax tight under Lipschitz exposure response. We also prove finite-catalog approximation and a robust selector theorem with excess-risk control, exact recovery under separation, and certified shortlists when the risk surface is flat. Empirically, the same selector gives different recommendations across samples from public datasets. It selects user-randomization on Criteo ads with dimensionless robust risk 1.295, switchbacks on Open Bandit-bts/men with risk 2.105, and cluster-randomization on KuaiRand with risk 2.240. The Open Bandit case stresses known but uneven logging support, with propensities from 0.00006 to 0.594 and a 5.17% IPS effective-sample share. Overall, the paper contributes an interference-aware experiment design framework based on mechanism-robust design decisions, where the output is either a justified design choice or an uncertainty shortlist.