mlsanityData to 14 Oct 2026

Are Two Datasets Close Enough With Statistical Significance? A Kernel Distributional Closeness Testing Approach

arXiv 2507.12843 ↗First posted 17 Jul 2025cs.LG stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

Are two distributions close to each other with statistical significance? Distribution closeness testing (DCT) formalizes this question by testing whether the distance between a distribution pair is at least epsilon-far. Existing DCT methods mainly measure discrepancies between distribution pairs defined on discrete spaces, for example using total variation, which limits their application to complex data such as images. To extend DCT to more types of data, a natural idea is to introduce maximum mean discrepancy (MMD), a powerful measure of distributional discrepancy between complex distributions, into DCT scenarios. However, empirical results indicate that many distribution pairs can have the same MMD value despite having different norms in the same reproducing kernel Hilbert space (RKHS). These pairs may exhibit different finite-sample distinguishability and reflect different practical closeness levels, making MMD less informative for DCT. To mitigate this issue, we design a new measure of distributional discrepancy, norm-adaptive MMD (NAMMD), which scales the MMD value using the RKHS norms of distributions. Based on the asymptotic distribution of NAMMD, we propose NAMMD-based DCT to assess the closeness level of a distribution pair. Theoretically, we prove that NAMMD-based DCT has higher test power than MMD-based DCT while maintaining bounded type-I error. This is further validated by extensive experiments on multiple types of data, including synthetic noise and real images. Our code is available at https://github.com/zhijianzhouml/NAMMD.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 33 of its 81 references that are in the map. First posted on arXiv 17 Jul 2025.

Builds on

33 of 81 references.

References · 33

  1. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  2. 2021Convex optimizationcited by 2,515
  3. 2016Kernel Mean Embedding of Distributions: A Review and Beyondcited by 364
  4. 2016Revisiting Classifier Two-Sample Testscited by 268
  5. 2018Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shiftcited by 243
  6. 2015On Wasserstein Two Sample Testing and Related Families of Nonparametric Testscited by 166
  7. 2020Learning Deep Kernels for Non-Parametric Two-Sample Testscited by 130
  8. 2016Generative Models and Model Criticism via Optimized Maximum Mean Discrepancycited by 100
  9. 2016Interpretable Distribution Features with Maximum Testing Powercited by 63
  10. 2015Fast Two-Sample Testing with Analytic Representations of Probability Measurescited by 53
  11. 2020Maximum Mean Discrepancy Test is Aware of Adversarial Attackscited by 44
  12. 2016Classification accuracy as a proxy for two sample testingcited by 42
21 more
  1. 2013Optimal Algorithms for Testing Closeness of Discrete Distributionscited by 37
  2. 2015Optimal Testing for Properties of Distributionscited by 36
  3. 2023MMD-FUSE: Learning and Combining Kernels for Two-Sample Testing Without Data Splittingcited by 33
  4. 2020A Kernel Two-Sample Test for Functional Datacited by 28
  5. 2020Domain Generalization via Optimal Transport with Metric Similarity Learningcited by 27
  6. 2021Higher Order Kernel Mean Embeddings to Capture Filtrations of Stochastic Processescited by 27
  7. 2022Efficient Aggregated Kernel Tests using Incomplete U-statisticscited by 27
  8. 2019Classification Logit Two-sample Testing by Neural Networkscited by 26
  9. 2022A Permutation-free Kernel Two-Sample Testcited by 21
  10. 2022AutoML Two-Sample Testcited by 20
  11. 2022Spectral Regularized Kernel Two-Sample Testscited by 20
  12. 2021The Price of Tolerance in Distribution Testingcited by 14
  13. 2019On the Use of Random Forest for Two-Sample Testingcited by 13
  14. 2015Optimal Algorithms and Lower Bounds for Testing Closeness of Structured Distributionscited by 11
  15. 2019Unbiased estimators for the variance of MMD estimatorscited by 10
  16. 2023Testing Closeness of Multivariate Distributions via Ramsey Theorycited by 8
  17. 2024A Unified Data Representation Learning for Non-parametric Two-sample Testingcited by 6
  18. 2019Comparing distributions: ℓ1 geometry improves kernel two-sample testingcited by 5
  19. 2015Testing Closeness With Unequal Sized Samplescited by 4
  20. 2017Active Tolerant Testingcited by 4
  21. 2023A Model-free Closeness-of-influence Test for Features in Supervised Learningcited by 1

Built on by

Not cited yet.