# Experimentation Platforms Meet Reinforcement Learning: Bayesian Sequential Decision-Making for Continuous Monitoring

arXiv: https://arxiv.org/abs/2304.00420 · HTML: https://mlsanity.com/paper/2304.00420/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 61 references that are in the map.
- First posted on arXiv 2 Apr 2023.

## Builds on (most cited first, 20 of 20)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Learning Dexterous In-Hand Manipulation](https://mlsanity.com/paper/1808.00177/) (2018)
- [Deep Reinforcement Learning framework for Autonomous Driving](https://mlsanity.com/paper/1704.02532/) (2017)
- [Contextual Markov Decision Processes](https://mlsanity.com/paper/1502.02259/) (2015)
- [Multi-Task Reinforcement Learning with Context-based Representations](https://mlsanity.com/paper/2102.06177/) (2021)
- [Deep Adaptive Design: Amortizing Sequential Bayesian Experimental Design](https://mlsanity.com/paper/2103.02438/) (2021)
- [Implicit Deep Adaptive Design: Policy-Based Experimental Design without Likelihoods](https://mlsanity.com/paper/2111.02329/) (2021)
- [Sequential Bayesian optimal experimental design via approximate dynamic programming](https://mlsanity.com/paper/1604.08320/) (2016)
- [Self-Paced Context Evaluation for Contextual Reinforcement Learning](https://mlsanity.com/paper/2106.05110/) (2021)
- [Bayesian Sequential Optimal Experimental Design for Nonlinear Models Using Policy Gradient Reinforcement Learning](https://mlsanity.com/paper/2110.15335/) (2021)
- [CARL: A Benchmark for Contextual and Adaptive Reinforcement Learning](https://mlsanity.com/paper/2110.02102/) (2021)
- [Always Valid Inference: Bringing Sequential Analysis to A/B Testing](https://mlsanity.com/paper/1512.04922/) (2015)
- [Multi-Objective Model-based Reinforcement Learning for Infectious Disease Control](https://mlsanity.com/paper/2009.04607/) (2020)
- [A Bayesian Model for Online Activity Sample Sizes](https://mlsanity.com/paper/2111.12157/) (2021)
- [A Comparative Tutorial of Bayesian Sequential Design and Reinforcement Learning](https://mlsanity.com/paper/2205.04023/) (2022)
- [An introduction to the determination of the probability of a successful trial: Frequentist and Bayesian approaches](https://mlsanity.com/paper/2102.13550/) (2021)

## Built on by (most cited first, 3 of 3)

- [Designing Time Series Experiments in A/B Testing with Transformer Reinforcement Learning](https://mlsanity.com/paper/2602.01853/) (2026)
- [Effect Size Estimation for Duration Recommendation in Online Experiments: Leveraging Hierarchical Models and Objective Utility Approaches](https://mlsanity.com/paper/2312.12871/) (2023)
- [Improved prediction of future user activity in online A/B testing](https://mlsanity.com/paper/2402.03231/) (2024)

## Abstract (arXiv)

With the growing needs of online A/B testing to support the innovation in industry, the opportunity cost of running an experiment becomes non-negligible. Therefore, there is an increasing demand for an efficient continuous monitoring service that allows early stopping when appropriate. Classic statistical methods focus on hypothesis testing and are mostly developed for traditional high-stake problems such as clinical trials, while experiments at online service companies typically have very different features and focuses. Motivated by the real needs, in this paper, we introduce a novel framework that we developed in Amazon to maximize customer experience and control opportunity cost. We formulate the problem as a Bayesian optimal sequential decision making problem that has a unified utility function. We discuss extensively practical design choices and considerations. We further introduce how to solve the optimal decision rule via Reinforcement Learning and scale the solution. We show the effectiveness of this novel approach compared with existing methods via a large-scale meta-analysis on experiments in Amazon.