# Beyond Basic A/B testing: Improving Statistical Efficiency for Business Growth

arXiv: https://arxiv.org/abs/2505.08128 · HTML: https://mlsanity.com/paper/2505.08128/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 53 references that are in the map.
- First posted on arXiv 13 May 2025.

## Builds on (most cited first, 7 of 7)

- [MMD Aggregated Two-Sample Test](https://mlsanity.com/paper/2110.15073/) (2021)
- [Efficient Aggregated Kernel Tests using Incomplete U-statistics](https://mlsanity.com/paper/2206.09194/) (2022)
- [Scaling-up Empirical Risk Minimization: Optimization of Incomplete U-statistics](https://mlsanity.com/paper/1501.02629/) (2015)
- [All about sample-size calculations for A/B testing: Novel extensions and practical guide](https://mlsanity.com/paper/2305.16459/) (2023)
- [Truncated Rank-Based Tests for Two-Part Models with Excessive Zeros and Applications to Microbiome Data](https://mlsanity.com/paper/2110.05368/) (2021)
- [A doubly robust estimator for the Mann Whitney Wilcoxon Rank Sum Test when applied for causal inference in observational studies](https://mlsanity.com/paper/2403.06783/) (2024)
- [A multi-locus predictiveness curve and its summary assessment for genetic risk prediction](https://mlsanity.com/paper/2504.00024/) (2025)

## Built on by (most cited first, 1 of 1)

- [BanditLP: Large-Scale Stochastic Optimization for Personalized Recommendations](https://mlsanity.com/paper/2601.15552/) (2026)

## Abstract (arXiv)

The standard A/B testing approaches are mostly based on t-test in large scale industry applications. These standard approaches however suffers from low statistical power in business settings, due to nature of small sample-size or non-Gaussian distribution or return-on-investment (ROI) consideration. In this paper, we (i) show the statistical efficiency of using estimating equation and U statistics, which can address these issues separately; and (ii) propose a novel doubly robust generalized U that allows flexible definition of treatment effect, and can handles small samples, distribution robustness, ROI and confounding consideration in one framework. We provide theoretical results on asymptotics and efficiency bounds, together with insights on the efficiency gain from theoretical analysis. We further conduct comprehensive simulation studies, apply the methods to multiple real A/B tests at LinkedIn, and share results and learnings that are broadly useful.