# Experimental Design in Two-Sided Platforms: An Analysis of Bias

arXiv: https://arxiv.org/abs/2002.05670 · HTML: https://mlsanity.com/paper/2002.05670/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Feb 2020.

## Built on by (most cited first, 30 of 39)

- [Markovian Interference in Experiments](https://mlsanity.com/paper/2206.02371/) (2022)
- [Treatment Effect Risk: Bounds and Inference](https://mlsanity.com/paper/2201.05893/) (2022)
- [Correcting for Interference in Experiments: A Case Study at Douyin](https://mlsanity.com/paper/2305.02542/) (2023)
- [Multi-Armed Bandits with Interference](https://mlsanity.com/paper/2402.01845/) (2024)
- [Policy Evaluation for Temporal and/or Spatial Dependent Experiments](https://mlsanity.com/paper/2202.10887/) (2022)
- [Causal Message Passing for Experiments with Unknown and General Network Interference](https://mlsanity.com/paper/2311.08340/) (2023)
- [Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experiments](https://mlsanity.com/paper/2403.17285/) (2024)
- [Speed Up the Cold-Start Learning in Two-Sided Bandits with Many Arms](https://mlsanity.com/paper/2210.00340/) (2022)
- [Higher-Order Causal Message Passing for Experimentation with Complex Interference](https://mlsanity.com/paper/2411.00945/) (2024)
- [Double Machine Learning for Causal Inference under Shared-State Interference](https://mlsanity.com/paper/2504.08836/) (2025)
- [Choosing the Better Bandit Algorithm under Data Sharing: When Do A/B Experiments Work?](https://mlsanity.com/paper/2507.11891/) (2025)
- [A Two-Part Machine Learning Approach to Characterizing Network Interference in A/B Testing](https://mlsanity.com/paper/2308.09790/) (2023)
- [Tackling Interference Induced by Data Training Loops in A/B Tests: A Weighted Training Approach](https://mlsanity.com/paper/2310.17496/) (2023)
- [Clustered Switchback Designs for Experimentation Under Spatio-temporal Interference](https://mlsanity.com/paper/2312.15574/) (2023)
- [Learning soft interventions in complex equilibrium systems](https://mlsanity.com/paper/2112.05729/) (2021)
- [Policy Learning with Competing Agents](https://mlsanity.com/paper/2204.01884/) (2022)
- [Network Synthetic Interventions: A Causal Framework for Panel Data Under Network Interference](https://mlsanity.com/paper/2210.11355/) (2022)
- [Cascade-based Randomization for Inferring Causal Effects under Diffusion Interference](https://mlsanity.com/paper/2405.12340/) (2024)
- [Causal Deepsets for Off-policy Evaluation under Spatial or Spatio-temporal Interferences](https://mlsanity.com/paper/2407.17910/) (2024)
- [Decision Support for Marketplace Policies under Incomplete Evidence: From Replay to Launch Readiness](https://mlsanity.com/paper/2605.12840/) (2026)
- [Differences-in-Neighbors for Network Interference in Experiments](https://mlsanity.com/paper/2503.02271/) (2025)
- [On Evolution-Based Models for Experimentation Under Interference](https://mlsanity.com/paper/2511.21675/) (2025)
- [Designing Time Series Experiments in A/B Testing with Transformer Reinforcement Learning](https://mlsanity.com/paper/2602.01853/) (2026)
- [Design Experiments to Compare Multi-armed Bandit Algorithms](https://mlsanity.com/paper/2603.05919/) (2026)
- [Support-aware offline policy selection for advertising marketplaces](https://mlsanity.com/paper/2605.21736/) (2026)
- [Choosing Online Experiment Designs under Interference in Ads, Recommendations, and Member-Experience Systems](https://mlsanity.com/paper/2605.25290/) (2026)
- [GUSH3R: Everyone Everywhere All at Once as Gaussians](https://mlsanity.com/paper/2607.05243/) (2026)
- [The Price of Isolation: Estimating the Ecosystem Cost of Symmetric Two-Sided A/B Testing](https://mlsanity.com/paper/2608.04432/) (2026)
- [Near-Optimal Experimental Design Under the Budget Constraint in Online Platforms](https://mlsanity.com/paper/2302.05005/) (2023)
- [Causal Estimation of User Learning in Personalized Systems](https://mlsanity.com/paper/2306.00485/) (2023)

## Abstract (arXiv)

We develop an analytical framework to study experimental design in two-sided marketplaces. Many of these experiments exhibit interference, where an intervention applied to one market participant influences the behavior of another participant. This interference leads to biased estimates of the treatment effect of the intervention. We develop a stochastic market model and associated mean field limit to capture dynamics in such experiments, and use our model to investigate how the performance of different designs and estimators is affected by marketplace interference effects. Platforms typically use two common experimental designs: demand-side ("customer") randomization (CR) and supply-side ("listing") randomization (LR), along with their associated estimators. We show that good experimental design depends on market balance: in highly demand-constrained markets, CR is unbiased, while LR is biased; conversely, in highly supply-constrained markets, LR is unbiased, while CR is biased. We also introduce and study a novel experimental design based on two-sided randomization (TSR) where both customers and listings are randomized to treatment and control. We show that appropriate choices of TSR designs can be unbiased in both extremes of market balance, while yielding relatively low bias in intermediate regimes of market balance.