# Tackling Interference Induced by Data Training Loops in A/B Tests: A Weighted Training Approach

arXiv: https://arxiv.org/abs/2310.17496 · HTML: https://mlsanity.com/paper/2310.17496/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 69 references that are in the map.
- First posted on arXiv 26 Oct 2023.

## Builds on (most cited first, 30 of 33)

- [How Algorithmic Confounding in Recommendation Systems Increases Homogeneity and Decreases Utility](https://mlsanity.com/paper/1710.11214/) (2017)
- [Feedback Loop and Bias Amplification in Recommender Systems](https://mlsanity.com/paper/2007.13019/) (2020)
- [Graph cluster randomization: network exposure to multiple universes](https://mlsanity.com/paper/1305.6979/) (2013)
- [Design and analysis of experiments in networks: Reducing bias from interference](https://mlsanity.com/paper/1404.7530/) (2014)
- [Experimental Design in Two-Sided Platforms: An Analysis of Bias](https://mlsanity.com/paper/2002.05670/) (2020)
- [Design and Analysis of Switchback Experiments](https://mlsanity.com/paper/2009.00148/) (2020)
- [Interference, Bias, and Variance in Two-Sided Marketplace Experimentation: Guidance for Platforms](https://mlsanity.com/paper/2104.12222/) (2021)
- [Markovian Interference in Experiments](https://mlsanity.com/paper/2206.02371/) (2022)
- [Multiple Randomization Designs: Estimation and Inference with Interference](https://mlsanity.com/paper/2112.13495/) (2021)
- [Estimating Total Treatment Effect in Randomized Experiments with Unknown Network Structure](https://mlsanity.com/paper/2205.12803/) (2022)
- [Switchback Experiments under Geometric Mixing](https://mlsanity.com/paper/2209.00197/) (2022)
- [Experimenting in Equilibrium](https://mlsanity.com/paper/1903.02124/) (2019)
- [Treatment Effects in Market Equilibrium](https://mlsanity.com/paper/2109.11647/) (2021)
- [Adaptive Experimental Design with Temporal Interference: A Maximum Likelihood Approach](https://mlsanity.com/paper/2006.05591/) (2020)
- [Randomized Graph Cluster Randomization](https://mlsanity.com/paper/2009.02297/) (2020)
- [Weak Signal Asymptotics for Sequentially Randomized Experiments](https://mlsanity.com/paper/2101.09855/) (2021)
- [Breaking Feedback Loops in Recommender Systems with Causal Inference](https://mlsanity.com/paper/2207.01616/) (2022)
- [Optimal Experimental Design for Staggered Rollouts](https://mlsanity.com/paper/1911.03764/) (2019)
- [Reducing Interference Bias in Online Marketplace Pricing Experiments](https://mlsanity.com/paper/2004.12489/) (2020)
- [Correcting for Interference in Experiments: A Case Study at Douyin](https://mlsanity.com/paper/2305.02542/) (2023)
- [A/B Testing for Recommender Systems in a Two-sided Marketplace](https://mlsanity.com/paper/2106.00762/) (2021)
- [Design and Analysis of Bipartite Experiments under a Linear Exposure-Response Model](https://mlsanity.com/paper/2103.06392/) (2021)
- [Reducing Symbiosis Bias Through Better A/B Tests of Recommendation Algorithms](https://mlsanity.com/paper/2309.07107/) (2023)
- [Limiting Bias from Test-Control Interference in Online Marketplace Experiments](https://mlsanity.com/paper/2004.12162/) (2020)
- [Evaluating Online Bandit Exploration In Large-Scale Recommender System](https://mlsanity.com/paper/2304.02572/) (2023)
- [Experimenting under Stochastic Congestion](https://mlsanity.com/paper/2302.12093/) (2023)
- [Randomization and The Pernicious Effects of Limited Budgets on Auction Experiments](https://mlsanity.com/paper/1605.09171/) (2016)
- [Correcting the User Feedback-Loop Bias for Recommendation Systems](https://mlsanity.com/paper/2109.06037/) (2021)
- [Reducing Marketplace Interference Bias Via Shadow Prices](https://mlsanity.com/paper/2205.02274/) (2022)
- [Modeling Interference Using Experiment Roll-out](https://mlsanity.com/paper/2305.10728/) (2023)

## Built on by (most cited first, 3 of 3)

- [Choosing the Better Bandit Algorithm under Data Sharing: When Do A/B Experiments Work?](https://mlsanity.com/paper/2507.11891/) (2025)
- [Design Experiments to Compare Multi-armed Bandit Algorithms](https://mlsanity.com/paper/2603.05919/) (2026)
- [BAFF: Bid-Aware Filter Family for Mitigating Training Data Interference in RTB A/B Tests](https://mlsanity.com/paper/2609.08725/) (2026)

## Abstract (arXiv)

In modern recommendation systems, the standard pipeline involves training machine learning models on historical data to predict user behaviors and improve recommendations continuously. However, these data training loops can introduce interference in A/B tests, where data generated by control and treatment algorithms, potentially with different distributions, are combined. To address these challenges, we introduce a novel approach called weighted training. This approach entails training a model to predict the probability of each data point appearing in either the treatment or control data and subsequently applying weighted losses during model training. We demonstrate that this approach achieves the least variance among all estimators that do not cause shifts in the training distributions. Through simulation studies, we demonstrate the lower bias and variance of our approach compared to other methods.