# t-Testing the Waters: Empirically Validating Assumptions for Reliable A/B-Testing

arXiv: https://arxiv.org/abs/2502.04793 · HTML: https://mlsanity.com/paper/2502.04793/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 26 references that are in the map.
- First posted on arXiv 7 Feb 2025.

## Builds on (most cited first, 7 of 7)

- [Machine Learning for Variance Reduction in Online Experiments](https://mlsanity.com/paper/2106.07263/) (2021)
- [Choosing a Proxy Metric from Past Experiments](https://mlsanity.com/paper/2309.07893/) (2023)
- [Variance Reduction in Ratio Metrics for Efficient Online Experiments](https://mlsanity.com/paper/2401.04062/) (2024)
- [Learning Metrics that Maximise Power for Accelerated A/B-Tests](https://mlsanity.com/paper/2402.03915/) (2024)
- [Abandon Statistical Significance](https://mlsanity.com/paper/1709.07588/) (2017)
- [A Common Misassumption in Online Experiments with Machine Learning Models](https://mlsanity.com/paper/2304.10900/) (2023)
- [Powerful A/B-Testing Metrics and Where to Find Them](https://mlsanity.com/paper/2407.20665/) (2024)

## Built on by (most cited first, 3 of 3)

- [Behavioural Effects of Agentic Messaging: A Case Study on a Financial Service Application](https://mlsanity.com/paper/2512.17462/) (2025)
- [Variance Reduction for Heavy-Tailed Monetization Metrics in Ranking Experiments via Post-Stratification](https://mlsanity.com/paper/2606.04110/) (2026)
- [Accelerating A/B-Tests with Counterfactual Estimation: Reducing Variance through Policy Overlap](https://mlsanity.com/paper/2607.14604/) (2026)

## Abstract (arXiv)

A/B-tests are a cornerstone of experimental design on the web, with wide-ranging applications and use-cases. The statistical t-test comparing differences in means is the most commonly used method for assessing treatment effects, often justified through the Central Limit Theorem (CLT). The CLT ascertains that, as the sample size grows, the sampling distribution of the Average Treatment Effect converges to normality, making the t-test valid for sufficiently large sample sizes. When outcome measures are skewed or non-normal, quantifying what "sufficiently large" entails is not straightforward. To ensure that confidence intervals maintain proper coverage and that p-values accurately reflect the false positive rate, it is critical to validate this normality assumption. We propose a practical method to test this, by analysing repeatedly resampled A/A-tests. When the normality assumption holds, the resulting p-value distribution should be uniform, and this property can be tested using the Kolmogorov-Smirnov test. This provides an efficient and effective way to empirically assess whether the t-test's assumptions are met, and the A/B-test is valid. We demonstrate our methodology and highlight how it helps to identify scenarios prone to inflated Type-I errors. Our approach provides a practical framework to ensure and improve the reliability and robustness of A/B-testing practices.