mlsanityData to 7 Oct 2026

AutoML Two-Sample Test

arXiv 2206.08843 ↗First posted 17 Jun 2022cs.LG stat.MLcited by 19

Not on the map. Why?

Abstract (arXiv)

Two-sample tests are important in statistics and machine learning, both as tools for scientific discovery as well as to detect distribution shifts. This led to the development of many sophisticated test procedures going beyond the standard supervised learning frameworks, whose usage can require specialized knowledge about two-sample testing. We use a simple test that takes the mean discrepancy of a witness function as the test statistic and prove that minimizing a squared loss leads to a witness with optimal testing power. This allows us to leverage recent advancements in AutoML. Without any user input about the problems at hand, and using the same method for all our experiments, our AutoML two-sample test achieves competitive performance on a diverse distribution shift benchmark as well as on challenging two-sample testing problems. We provide an implementation of the AutoML two-sample test in the Python package autotst.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 20 of its 53 references that are in the map. First posted on arXiv 17 Jun 2022.

Builds on

20 of 53 references.

References · 20

  1. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  2. 2019AutoML: A Survey of the State-of-the-Artcited by 388
  3. 2020AutoGluon-Tabular: Robust and Accurate AutoML for Structured Datacited by 368
  4. 2018Detecting and Correcting for Label Shift with Black Box Predictorscited by 364
  5. 2016Revisiting Classifier Two-Sample Testscited by 258
  6. 2018Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shiftcited by 240
  7. 2020Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learningcited by 174
  8. 2020Learning Deep Kernels for Non-Parametric Two-Sample Testscited by 129
  9. 2016Generative Models and Model Criticism via Optimized Maximum Mean Discrepancycited by 99
  10. 2021MMD Aggregated Two-Sample Testcited by 74
  11. 2016Interpretable Distribution Features with Maximum Testing Powercited by 63
  12. 2015Fast Two-Sample Testing with Analytic Representations of Probability Measurescited by 53
8 more
  1. 2016Classification accuracy as a proxy for two sample testingcited by 39
  2. 2019Two-sample Testing Using Deep Learningcited by 26
  3. 2019Classification Logit Two-sample Testing by Neural Networkscited by 25
  4. 2021A Witness Two-Sample Testcited by 20
  5. 2020Learning Kernel Tests Without Data Splittingcited by 19
  6. 2017On the Effectiveness of Least Squares Generative Adversarial Networkscited by 18
  7. 2021Meta Two-Sample Testing: Learning Kernels for Testing with Limited Datacited by 18
  8. 2019On the Use of Random Forest for Two-Sample Testingcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 3; 2025: 7; 2026: 4.

19 papers.

Sort:

By area:

  1. 2023MMD-FUSE: Learning and Combining Kernels for Two-Sample Testing Without Data Splittingcited by 33
  2. 2023Sequential Predictive Two-Sample and Independence Testingcited by 20
  3. 2024A Practical Guide to Sample-based Statistical Distances for Evaluating Generative Models in Sciencecited by 14
  4. 2023The Representation Jensen-Shannon Divergencecited by 9
  5. 2023Variable Selection for Kernel Two-Sample Testscited by 7
  6. 2025DUAL: Learning Diverse Kernels for Aggregated Two-sample and Independence Testingcited by 6
  7. 2024A Unified Data Representation Learning for Non-parametric Two-sample Testingcited by 5
  8. 2025A Scalable Nystrom-Based Kernel Two-Sample Test with Permutationscited by 4
  9. 2026Regularized f-Divergence Kernel Testscited by 2
  10. 2023Deep Hypothesis Tests Detect Clinically Relevant Subgroup Shifts in Medical Imagescited by 1
  11. 2025Anchor-based Maximum Discrepancy for Relative Similarity Testingcited by 1
  12. 2025Sparse, self-organizing ensembles of local kernels detect rare statistical anomaliescited by 1
7 more
  1. 2025Maximum Mean Discrepancy with Unequal Sample Sizes via Generalized U-Statisticscited by 1
  2. 2026Zero-Flow Two-Sample Testscited by 1
  3. 2024Machine Learning for Two-Sample Testing under Right-Censored Data: A Simulation Studycited by 0
  4. 2025Boosting Statistic Learning with Synthetic Data from Pretrained Large Modelscited by 0
  5. 2025Are Two Datasets Close Enough With Statistical Significance? A Kernel Distributional Closeness Testing Approachcited by 0
  6. 2026A nonparametric two-sample test using a parametric integral probability metriccited by 0
  7. 2026Dimension-Calibrated Unexplained Mass: An Interpretable Drift Statistic for Contamination Monitoring in Data Streamscited by 0