# Validation of massively-parallel adaptive testing using dynamic control matching

arXiv: https://arxiv.org/abs/2305.01334 · HTML: https://mlsanity.com/paper/2305.01334/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 18 references that are in the map.
- First posted on arXiv 2 May 2023.

## Builds on (most cited first, 9 of 9)

- [Generalization in Adaptive Data Analysis and Holdout Reuse](https://mlsanity.com/paper/1506.02629/) (2015)
- [Design and Analysis of Switchback Experiments](https://mlsanity.com/paper/2009.00148/) (2020)
- [Interference, Bias, and Variance in Two-Sided Marketplace Experimentation: Guidance for Platforms](https://mlsanity.com/paper/2104.12222/) (2021)
- [Switchback Experiments under Geometric Mixing](https://mlsanity.com/paper/2209.00197/) (2022)
- [Confounder Adjustment in Multiple Hypothesis Testing](https://mlsanity.com/paper/1508.04178/) (2015)
- [Novelty and Primacy: A Long-Term Estimator for Online Experiments](https://mlsanity.com/paper/2102.12893/) (2021)
- [Synthetic Controls in Action](https://mlsanity.com/paper/2203.06279/) (2022)
- [Methodology to Create Analysis-Naive Holdout Records as well as Train and Test Records for Machine Learning Analyses in Healthcare](https://mlsanity.com/paper/2205.03987/) (2022)
- [Bayesian and Frequentist Inference for Synthetic Controls](https://mlsanity.com/paper/2206.01779/) (2022)

## Abstract (arXiv)

A/B testing is a widely-used paradigm within marketing optimization because it promises identification of causal effects and because it is implemented out of the box in most messaging delivery software platforms. Modern businesses, however, often run many A/B/n tests at the same time and in parallel, and package many content variations into the same messages, not all of which are part of an explicit test. Whether as the result of many teams testing at the same time, or as part of a more sophisticated reinforcement learning (RL) approach that continuously adapts tests and test condition assignment based on previous results, dynamic parallel testing cannot be evaluated the same way traditional A/B tests are evaluated. This paper presents a method for disentangling the causal effects of the various tests under conditions of continuous test adaptation, using a matched-synthetic control group that adapts alongside the tests.