# Interference, Bias, and Variance in Two-Sided Marketplace Experimentation: Guidance for Platforms

arXiv: https://arxiv.org/abs/2104.12222 · HTML: https://mlsanity.com/paper/2104.12222/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Apr 2021.

## Built on by (most cited first, 22 of 22)

- [Bayesian Optimization of Function Networks](https://mlsanity.com/paper/2112.15311/) (2021)
- [Correcting for Interference in Experiments: A Case Study at Douyin](https://mlsanity.com/paper/2305.02542/) (2023)
- [Data-Driven Switchback Experiments: Theoretical Tradeoffs and Empirical Bayes Designs](https://mlsanity.com/paper/2406.06768/) (2024)
- [Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experiments](https://mlsanity.com/paper/2403.17285/) (2024)
- [Prediction-Guided Active Experiments](https://mlsanity.com/paper/2411.12036/) (2024)
- [Balancing Interference and Correlation in Spatial Experimental Designs: A Causal Graph Cut Approach](https://mlsanity.com/paper/2505.20130/) (2025)
- [Choosing the Better Bandit Algorithm under Data Sharing: When Do A/B Experiments Work?](https://mlsanity.com/paper/2507.11891/) (2025)
- [Off-Policy Evaluation with Policy-Dependent Optimization Response](https://mlsanity.com/paper/2202.12958/) (2022)
- [A Two-Part Machine Learning Approach to Characterizing Network Interference in A/B Testing](https://mlsanity.com/paper/2308.09790/) (2023)
- [Tackling Interference Induced by Data Training Loops in A/B Tests: A Weighted Training Approach](https://mlsanity.com/paper/2310.17496/) (2023)
- [Leveraging heterogeneous spillover in maximizing contextual bandit rewards](https://mlsanity.com/paper/2310.10259/) (2023)
- [Decision Support for Marketplace Policies under Incomplete Evidence: From Replay to Launch Readiness](https://mlsanity.com/paper/2605.12840/) (2026)
- [Differences-in-Neighbors for Network Interference in Experiments](https://mlsanity.com/paper/2503.02271/) (2025)
- [Designing Time Series Experiments in A/B Testing with Transformer Reinforcement Learning](https://mlsanity.com/paper/2602.01853/) (2026)
- [Support-aware offline policy selection for advertising marketplaces](https://mlsanity.com/paper/2605.21736/) (2026)
- [Choosing Online Experiment Designs under Interference in Ads, Recommendations, and Member-Experience Systems](https://mlsanity.com/paper/2605.25290/) (2026)
- [The Price of Isolation: Estimating the Ecosystem Cost of Symmetric Two-Sided A/B Testing](https://mlsanity.com/paper/2608.04432/) (2026)
- [Validation of massively-parallel adaptive testing using dynamic control matching](https://mlsanity.com/paper/2305.01334/) (2023)
- [A/B testing under Interference with Partial Network Information](https://mlsanity.com/paper/2404.10547/) (2024)
- [Towards Measuring Sell Side Outcomes in Buy Side Marketplace Experiments using In-Experiment Bipartite Graph](https://mlsanity.com/paper/2409.04174/) (2024)
- [Unbiased Platform-Level Causal Estimation for Search Systems: A Competitive Isolation PSM-DID Framework](https://mlsanity.com/paper/2511.01329/) (2025)
- [Learning to target with network interference](https://mlsanity.com/paper/2605.27794/) (2026)

## Abstract (arXiv)

Two-sided marketplace platforms often run experiments to test the effect of an intervention before launching it platform-wide. A typical approach is to randomize individuals into the treatment group, which receives the intervention, and the control group, which does not. The platform then compares the performance in the two groups to estimate the effect if the intervention were launched to everyone. We focus on two common experiment types, where the platform randomizes individuals either on the supply side or on the demand side. The resulting estimates of the treatment effect in these experiments are typically biased: because individuals in the market compete with each other, individuals in the treatment group affect those in the control group and vice versa, creating interference. We develop a simple tractable market model to study bias and variance in these experiments with interference. We focus on two choices available to the platform: (1) Which side of the platform should it randomize on (supply or demand)? (2) What proportion of individuals should be allocated to treatment? We find that both choices affect the bias and variance of the resulting estimators but in different ways. The bias-optimal choice of experiment type depends on the relative amounts of supply and demand in the market, and we discuss how a platform can use market data to select the experiment type. Importantly, we find in many circumstances, choosing the bias-optimal experiment type has little effect on variance. On the other hand, the choice of treatment proportion can induce a bias-variance tradeoff, where the bias-minimizing proportion increases variance. We discuss how a platform can navigate this tradeoff and best choose the treatment proportion, using a combination of modeling as well as contextual knowledge about the market, the risk of the intervention, and reasonable effect sizes of the intervention.