# Limiting Bias from Test-Control Interference in Online Marketplace Experiments

arXiv: https://arxiv.org/abs/2004.12162 · HTML: https://mlsanity.com/paper/2004.12162/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Apr 2020.

## Built on by (most cited first, 7 of 7)

- [Correcting for Interference in Experiments: A Case Study at Douyin](https://mlsanity.com/paper/2305.02542/) (2023)
- [Tackling Interference Induced by Data Training Loops in A/B Tests: A Weighted Training Approach](https://mlsanity.com/paper/2310.17496/) (2023)
- [Decision Support for Marketplace Policies under Incomplete Evidence: From Replay to Launch Readiness](https://mlsanity.com/paper/2605.12840/) (2026)
- [Differences-in-Neighbors for Network Interference in Experiments](https://mlsanity.com/paper/2503.02271/) (2025)
- [Support-aware offline policy selection for advertising marketplaces](https://mlsanity.com/paper/2605.21736/) (2026)
- [Choosing Online Experiment Designs under Interference in Ads, Recommendations, and Member-Experience Systems](https://mlsanity.com/paper/2605.25290/) (2026)
- [ProfiLLM: Utility-Aligned Agentic User Profiling for Industrial Ride-Hailing Dispatch](https://mlsanity.com/paper/2606.18803/) (2026)

## Abstract (arXiv)

In an A/B test, the typical objective is to measure the total average treatment effect (TATE), which measures the difference between the average outcome if all users were treated and the average outcome if all users were untreated. However, a simple difference-in-means estimator will give a biased estimate of the TATE when outcomes of control units depend on the outcomes of treatment units, an issue we refer to as test-control interference. Using a simulation built on top of data from Airbnb, this paper considers the use of methods from the network interference literature for online marketplace experimentation. We model the marketplace as a network in which an edge exists between two sellers if their goods substitute for one another. We then simulate seller outcomes, specifically considering a "status quo" context and "treatment" context that forces all sellers to lower their prices. We use the same simulation framework to approximate TATE distributions produced by using blocked graph cluster randomization, exposure modeling, and the Hajek estimator for the difference in means. We find that while blocked graph cluster randomization reduces the bias of the naive difference-in-means estimator by as much as 62%, it also significantly increases the variance of the estimator. On the other hand, the use of more sophisticated estimators produces mixed results. While some provide (small) additional reductions in bias and small reductions in variance, others lead to increased bias and variance. Overall, our results suggest that experiment design and analysis techniques from the network experimentation literature are promising tools for reducing bias due to test-control interference in marketplace experiments.