mlsanityData to 14 Oct 2026

Multi-Metric Adaptive Experimental Design Under a Fixed Budget with Validation

arXiv 2506.03062 ↗First posted 3 Jun 2025cs.LG stat.MLcited by 1

Not on the map. Why?

Abstract (arXiv)

A/B tests in online experiments face statistical power challenges when testing multiple candidates simultaneously, while adaptive experimental designs (AED) alone fall short in inferring experiment statistics such as the average treatment effect, especially with many metrics (e.g., revenue, safety) and heterogeneous variances. This paper proposes a fixed-budget multi-metric AED framework with a two-phase structure: an adaptive exploration phase to identify the best treatment, and a validation phase with an A/B test to verify the treatment's quality and infer statistics. We propose SHRVar, which generalizes sequential halving (SH) with a novel relative-variance-based sampling and an elimination strategy built on reward z values. It achieves a provable error probability that decreases exponentially, where the exponent H3 generalizes the complexity measure for SH and SHVar with homogeneous and heterogeneous variances, respectively. Numerical experiments demonstrate its performance and robustness.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 67 of its 114 references that are in the map. First posted on arXiv 3 Jun 2025.

Builds on

67 of 114 references.

References · 67

  1. 2014On the Complexity of Best Arm Identification in Multi-Armed Bandit Modelscited by 286
  2. 2016Optimal Best Arm Identification with Fixed Confidencecited by 239
  3. 2013Bandits with Knapsackscited by 164
  4. 2019Solving a Class of Non-Convex Min-Max Games Using Iterative First Order Methodscited by 148
  5. 2016Bayesian Optimization with Safety Constraints: Safe and Automatic Parameter Tuning in Roboticscited by 134
  6. 2016Simple Bayesian Algorithms for Best Arm Identificationcited by 117
  7. 2014Bayesian Optimization with Unknown Constraintscited by 109
  8. 2018Stagewise Safe Bayesian Optimization with Gaussian Processescited by 79
  9. 2015Safe Controller Optimization for Quadrotors with Gaussian Processescited by 78
  10. 2019Adaptive and Safe Bayesian Optimization in High Dimensions via One-Dimensional Subspacescited by 73
  11. 2020Stochastic Bandits with Linear Constraintscited by 68
  12. 2019Pure Exploration with Multiple Correct Answerscited by 64
38 more
  1. 2016An optimal algorithm for the Thresholding Bandit Problemcited by 62
  2. 2019Safe Exploration for Interactive Machine Learningcited by 51
  3. 2015A General Framework for Constrained Bayesian Optimization using Information-based Searchcited by 50
  4. 2016Truncated Variance Reduction: A Unified Approach to Bayesian Optimization and Level-Set Estimationcited by 40
  5. 2019The True Sample Complexity of Identifying Good Armscited by 37
  6. 2017Robust Optimization for Non-Convex Objectivescited by 35
  7. 2022Top Two Algorithms Revisitedcited by 34
  8. 2017Good Arm Identification via Bandit Feedbackcited by 30
  9. 2021An Efficient Pessimistic-Optimistic Algorithm for Stochastic Linear Bandits with General Constraintscited by 29
  10. 2017Nearly Instance Optimal Sample Complexity Bounds for Top-k Arm Selectioncited by 28
  11. 2021Best Arm Identification with Safety Constraintscited by 25
  12. 2022Revisiting Simple Regret: Fast Rates for Returning a Good Armcited by 25
  13. 2023Semiparametric Efficient Inference in Adaptive Experimentscited by 23
  14. 2018Ordered Preference Elicitation Strategies for Supporting Multi-Objective Decision Makingcited by 18
  15. 2021A/B/n Testing with Control in the Presence of Subpopulationscited by 17
  16. 2021Nearly Optimal Algorithms for Level Set Estimationcited by 17
  17. 2022Almost Optimal Variance-Constrained Best Arm Identificationcited by 16
  18. 2022Active Learning with Safety Constraintscited by 16
  19. 2018Robust Super-Level Set Estimation using Gaussian Processescited by 15
  20. 2025Sequential Learning of the Pareto Front for Multi-objective Banditscited by 15
  21. 2014On the Complexity of A/B Testingcited by 14
  22. 2016Pure Exploration of Multi-armed Bandit Under Matroid Constraintscited by 14
  23. 2018Sequential Test for the Lowest Mean: From Thompson to Murphy Samplingcited by 14
  24. 2022Bayesian Fixed-Budget Best-Arm Identificationcited by 13
  25. 2024Cost Aware Best Arm Identificationcited by 13
  26. 2021Vector Optimization with Stochastic Bandit Feedbackcited by 12
  27. 2017Thresholding Bandits with Augmented UCBcited by 11
  28. 2022Interactively Learning Preference Constraints in Linear Banditscited by 11
  29. 2023Optimal Treatment Allocation for Efficient Policy Evaluation in Sequential Decision Makingcited by 11
  30. 2019Multiscale Gaussian Process Level Set Estimationcited by 10
  31. 2019Bayesian Experimental Design for Finding Reliable Level Set under Input Uncertaintycited by 10
  32. 2023Pure Exploration in Bandits with Linear Constraintscited by 10
  33. 2023Fast and Regret Optimal Best Arm Identification: Fundamental Limits and Low-Complexity Algorithmscited by 10
  34. 2024Multi-Objective Recommendation via Multivariate Policy Learningcited by 10
  35. 2020A New Perspective on Pool-Based Active Classification and False-Discovery Controlcited by 9
  36. 2022Constrained Pure Exploration Multi-Armed Bandits with a Fixed Budgetcited by 9
  37. 2022Information-Theoretic Safe Exploration with Gaussian Processescited by 9
  38. 2024Optimal Multi-Fidelity Best-Arm Identificationcited by 9

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Designing Time Series Experiments in A/B Testing with Transformer Reinforcement Learningcited by 1