# Reducing Symbiosis Bias Through Better A/B Tests of Recommendation Algorithms

arXiv: https://arxiv.org/abs/2309.07107 · HTML: https://mlsanity.com/paper/2309.07107/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Sep 2023.

## Built on by (most cited first, 8 of 8)

- [Double Machine Learning for Causal Inference under Shared-State Interference](https://mlsanity.com/paper/2504.08836/) (2025)
- [Choosing the Better Bandit Algorithm under Data Sharing: When Do A/B Experiments Work?](https://mlsanity.com/paper/2507.11891/) (2025)
- [Tackling Interference Induced by Data Training Loops in A/B Tests: A Weighted Training Approach](https://mlsanity.com/paper/2310.17496/) (2023)
- [Bias Mitigation for AI-Feedback Loops in Recommender Systems: A Systematic Literature Review and Taxonomy](https://mlsanity.com/paper/2509.00109/) (2025)
- [Design Experiments to Compare Multi-armed Bandit Algorithms](https://mlsanity.com/paper/2603.05919/) (2026)
- [Fast A/B/n Testing: Exact Multi-Policy Comparison via Tree-Coupled Feedback Sharing](https://mlsanity.com/paper/2608.12831/) (2026)
- [Content Exploration Beyond the Feed: Creator Supply and the Shared Corpus](https://mlsanity.com/paper/2608.29430/) (2026)
- [BAFF: Bid-Aware Filter Family for Mitigating Training Data Interference in RTB A/B Tests](https://mlsanity.com/paper/2609.08725/) (2026)

## Abstract (arXiv)

It is increasingly common in digital environments to use A/B tests to compare the performance of recommendation algorithms. However, such experiments often violate the stable unit treatment value assumption (SUTVA), particularly SUTVA's "no hidden treatments" assumption, due to the shared data between algorithms being compared. This results in a novel form of bias, which we term "symbiosis bias," where the performance of each algorithm is influenced by the training data generated by its competitor. In this paper, we investigate three experimental designs--cluster-randomized, data-diverted, and user-corpus co-diverted experiments--aimed at mitigating symbiosis bias. We present a theoretical model of symbiosis bias and simulate the impact of each design in dynamic recommendation environments. Our results show that while each design reduces symbiosis bias to some extent, they also introduce new challenges, such as reduced training data in data-diverted experiments. We further validate the existence of symbiosis bias using data from a large-scale A/B test conducted on a global recommender system, demonstrating that symbiosis bias affects treatment effect estimates in the field. Our findings provide actionable insights for researchers and practitioners seeking to design experiments that accurately capture algorithmic performance without bias in treatment effect estimates introduced by shared data.