# Practical Batch Bayesian Sampling Algorithms for Online Adaptive Traffic Experimentation

arXiv: https://arxiv.org/abs/2305.14704 · HTML: https://mlsanity.com/paper/2305.14704/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 27 references that are in the map.
- First posted on arXiv 24 May 2023.

## Builds on (most cited first, 6 of 6)

- [Simple Bayesian Algorithms for Best Arm Identification](https://mlsanity.com/paper/1602.08448/) (2016)
- [Confidence Intervals for Policy Evaluation in Adaptive Experiments](https://mlsanity.com/paper/1911.02768/) (2019)
- [Why Adaptively Collected Data Have Negative Bias and How to Correct for It](https://mlsanity.com/paper/1708.01977/) (2017)
- [An Efficient Bandit Algorithm for Realtime Multivariate Optimization](https://mlsanity.com/paper/1810.09558/) (2018)
- [Shrinkage Estimators in Online Experiments](https://mlsanity.com/paper/1904.12918/) (2019)
- [Efficient Multivariate Bandit Algorithm with Path Planning](https://mlsanity.com/paper/1909.02705/) (2019)

## Built on by (most cited first, 1 of 1)

- [Sequential Causal Discovery with Noisy Language Model Priors](https://mlsanity.com/paper/2506.16234/) (2025)

## Abstract (arXiv)

Online controlled experiments have emerged as industry gold standard for assessing new web features. As new web algorithms proliferate, experimentation platform faces an increasing demand on the velocity of online experiments, which encourages adaptive traffic testing methods to speed up identifying best variant by efficiently allocating traffic. This paper proposed four Bayesian batch bandit algorithms (NB-TS, WB-TS, NB-TTTS, WB-TTTS) for eBay's experimentation platform, using summary batch statistics of a goal metric without incurring new engineering technical debts. The novel WB-TTTS, in particular, demonstrates as an efficient, trustworthy and robust alternative to fixed horizon A/B testing. Another novel contribution is to bring trustworthiness of best arm identification algorithms into evaluation criterion and highlight the existence of severe false positive inflation with equivalent best arms. To gain the trust of experimenters, experimentation platform must consider both efficiency and trustworthiness; However, to the best of authors' knowledge, trustworthiness as an important topic is rarely discussed. This paper shows that Bayesian bandits without neutral posterior reshaping, particularly naive Thompson sampling (NB-TS), are untrustworthy because they can always identify an arm as the best from equivalent best arms. To restore trustworthiness, a novel finding uncovers connections between convergence distribution of posterior optimal probabilities of equivalent best arms and neutral posterior reshaping, which controls false positives. Lastly, this paper presents lessons learned from eBay's experience, as well as thorough evaluations. We hope this work is useful to other industrial practitioners and inspires academic researchers interested in the trustworthiness of adaptive traffic experimentation.