PTED: A multi-dimensional two-sample test for scientific inference and generative machine learning

Two-sample tests are widely applicable in inference and generative modelling, yet users frequently fall back on heuristics and visual inspection due to lack of an accessible test that operates in multiple dimensions. I present Permutation Test using the Energy Distance (PTED), a Python implementation of a powerful two-sample test described in Sz'ekely and Rizzo [2004]. The test statistic is the energy distance, a metric on probability distributions that admits a natural sample estimator built from pairwise distances. A permutation test is run on energy distances to produce an exact two-sample test. Because the statistic depends on the data only through pairwise distances, the test applies in high dimensions, on learned feature representations, at large or small or imbalanced sample sizes, and to any data type on which a distance can be defined. An approximation to the formula allows PTED to scale linearly with both number of dimensions and samples while retaining most discriminative power. I highlight a number of instructive sensitivity tests on PTED alongside other multi-dimensional two-sample tests for comparison. PTED is the only method that is sensitive to all tests, though conventional Maximum Mean Discrepancy is identical for all but an over-fitting test.