# Classification accuracy as a proxy for two sample testing

arXiv: https://arxiv.org/abs/1602.02210 · HTML: https://mlsanity.com/paper/1602.02210/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Feb 2016.

## Built on by (most cited first, 30 of 42)

- [Testing for Outliers with Conformal p-values](https://mlsanity.com/paper/2104.08279/) (2021)
- [Adversarial random forests for density estimation and generative modeling](https://mlsanity.com/paper/2205.09435/) (2022)
- [A Witness Two-Sample Test](https://mlsanity.com/paper/2102.05573/) (2021)
- [Sequential Predictive Two-Sample and Independence Testing](https://mlsanity.com/paper/2305.00143/) (2023)
- [AutoML Two-Sample Test](https://mlsanity.com/paper/2206.08843/) (2022)
- [Meta Two-Sample Testing: Learning Kernels for Testing with Limited Data](https://mlsanity.com/paper/2106.07636/) (2021)
- [MMD-B-Fair: Learning Fair Representations with Statistical Testing](https://mlsanity.com/paper/2211.07907/) (2022)
- [E-Valuating Classifier Two-Sample Tests](https://mlsanity.com/paper/2210.13027/) (2022)
- [Boosting the Power of Kernel Two-Sample Tests](https://mlsanity.com/paper/2302.10687/) (2023)
- [Deep anytime-valid hypothesis testing](https://mlsanity.com/paper/2310.19384/) (2023)
- [Benchmarking the Fidelity and Utility of Synthetic Relational Data](https://mlsanity.com/paper/2410.03411/) (2024)
- [A Unified Data Representation Learning for Non-parametric Two-sample Testing](https://mlsanity.com/paper/2412.00613/) (2024)
- [Test for non-negligible adverse shifts](https://mlsanity.com/paper/2107.02990/) (2021)
- [The Lifecycle of a Statistical Model: Model Failure Detection, Identification, and Refitting](https://mlsanity.com/paper/2202.04166/) (2022)
- [General Frameworks for Conditional Two-Sample Testing](https://mlsanity.com/paper/2410.16636/) (2024)
- [Model-Agnostic Signal Discovery with Machine Learning: Bridging the Gap Between Theory and Practice](https://mlsanity.com/paper/2605.31103/) (2026)
- [A hypothesis-driven method based on machine learning for neuroimaging data analysis](https://mlsanity.com/paper/2202.04397/) (2022)
- [MAGDiff: Covariate Data Set Shift Detection via Activation Graphs of Deep Neural Networks](https://mlsanity.com/paper/2305.13271/) (2023)
- [Kernel-Based Testing for Single-Cell Differential Analysis](https://mlsanity.com/paper/2307.08509/) (2023)
- [Measuring the Dependency Gap: Diagnosing Inter-Column Fidelity in Tabular Generative Models](https://mlsanity.com/paper/2607.21636/) (2026)
- [Detecting Distributional Differences in Labeled Sequence Data with Application to Tropical Cyclone Satellite Imagery](https://mlsanity.com/paper/2202.02253/) (2022)
- [Conformal C2ST: Turning weak classifiers into strong two-sample tests](https://mlsanity.com/paper/2507.17026/) (2025)
- [Anchor-based Maximum Discrepancy for Relative Similarity Testing](https://mlsanity.com/paper/2510.10477/) (2025)
- [Maximum Mean Discrepancy with Unequal Sample Sizes via Generalized U-Statistics](https://mlsanity.com/paper/2512.13997/) (2025)
- [LOTTERY: Learning from Reference-Only Samples in Two-Sample Testing under Size Asymmetry](https://mlsanity.com/paper/2606.08460/) (2026)
- [Two-Sample Testing via Path-based Inference](https://mlsanity.com/paper/2610.05684/) (2026)
- [Deep Learning-Based Estimation and Goodness-of-Fit for Large-Scale Confirmatory Item Factor Analysis](https://mlsanity.com/paper/2109.09500/) (2021)
- [Identifying Distributional Differences in Convective Evolution Prior to Rapid Intensification in Tropical Cyclones](https://mlsanity.com/paper/2109.12029/) (2021)
- [Two-sample test based on Self-Organizing Maps](https://mlsanity.com/paper/2212.08960/) (2022)
- [A framework for paired-sample hypothesis testing for high-dimensional data](https://mlsanity.com/paper/2309.16274/) (2023)

## Abstract (arXiv)

When data analysts train a classifier and check if its accuracy is significantly different from chance, they are implicitly performing a two-sample test. We investigate the statistical properties of this flexible approach in the high-dimensional setting. We prove two results that hold for all classifiers in any dimensions: if its true error remains ε-better than chance for some ε>0 as d,n → ∞, then (a) the permutation-based test is consistent (has power approaching to one), (b) a computationally efficient test based on a Gaussian approximation of the null distribution is also consistent. To get a finer understanding of the rates of consistency, we study a specialized setting of distinguishing Gaussians with mean-difference δ and common (known or unknown) covariance Sigma, when d/n → c ∈ (0,∞). We study variants of Fisher's linear discriminant analysis (LDA) such as "naive Bayes" in a nontrivial regime when ε → 0 (the Bayes classifier has true accuracy approaching 1/2), and contrast their power with corresponding variants of Hotelling's test. Surprisingly, the expressions for their power match exactly in terms of n,d,δ,Sigma, and the LDA approach is only worse by a constant factor, achieving an asymptotic relative efficiency (ARE) of 1/√π for balanced samples. We also extend our results to high-dimensional elliptical distributions with finite kurtosis. Other results of independent interest include minimax lower bounds, and the optimality of Hotelling's test when d=o(n). Simulation results validate our theory, and we present practical takeaway messages along with natural open problems.