mlsanityData to 7 Oct 2026

Experimentation Platforms Meet Reinforcement Learning: Bayesian Sequential Decision-Making for Continuous Monitoring

arXiv 2304.00420 ↗First posted 2 Apr 2023cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

With the growing needs of online A/B testing to support the innovation in industry, the opportunity cost of running an experiment becomes non-negligible. Therefore, there is an increasing demand for an efficient continuous monitoring service that allows early stopping when appropriate. Classic statistical methods focus on hypothesis testing and are mostly developed for traditional high-stake problems such as clinical trials, while experiments at online service companies typically have very different features and focuses. Motivated by the real needs, in this paper, we introduce a novel framework that we developed in Amazon to maximize customer experience and control opportunity cost. We formulate the problem as a Bayesian optimal sequential decision making problem that has a unified utility function. We discuss extensively practical design choices and considerations. We further introduce how to solve the optimal decision rule via Reinforcement Learning and scale the solution. We show the effectiveness of this novel approach compared with existing methods via a large-scale meta-analysis on experiments in Amazon.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 20 of its 61 references that are in the map. First posted on arXiv 2 Apr 2023.

Builds on

20 of 61 references.

References · 20

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Trust Region Policy Optimizationcited by 2,503
  4. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  5. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  6. 2018Learning Dexterous In-Hand Manipulationcited by 643
  7. 2017Deep Reinforcement Learning framework for Autonomous Drivingcited by 224
  8. 2015Contextual Markov Decision Processescited by 213
  9. 2021Multi-Task Reinforcement Learning with Context-based Representationscited by 169
  10. 2021Deep Adaptive Design: Amortizing Sequential Bayesian Experimental Designcited by 103
  11. 2021Implicit Deep Adaptive Design: Policy-Based Experimental Design without Likelihoodscited by 61
  12. 2016Sequential Bayesian optimal experimental design via approximate dynamic programmingcited by 27
8 more
  1. 2021Self-Paced Context Evaluation for Contextual Reinforcement Learningcited by 25
  2. 2021Bayesian Sequential Optimal Experimental Design for Nonlinear Models Using Policy Gradient Reinforcement Learningcited by 22
  3. 2021CARL: A Benchmark for Contextual and Adaptive Reinforcement Learningcited by 21
  4. 2015Always Valid Inference: Bringing Sequential Analysis to A/B Testingcited by 20
  5. 2020Multi-Objective Model-based Reinforcement Learning for Infectious Disease Controlcited by 7
  6. 2021A Bayesian Model for Online Activity Sample Sizescited by 4
  7. 2022A Comparative Tutorial of Bayesian Sequential Design and Reinforcement Learningcited by 3
  8. 2021An introduction to the determination of the probability of a successful trial: Frequentist and Bayesian approachescited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 1; 2025: 0; 2026: 1.

3 papers.

  1. 2026Designing Time Series Experiments in A/B Testing with Transformer Reinforcement Learningcited by 1
  2. 2023Effect Size Estimation for Duration Recommendation in Online Experiments: Leveraging Hierarchical Models and Objective Utility Approachescited by 0
  3. 2024Improved prediction of future user activity in online A/B testingcited by 0