mlsanityData to 14 Oct 2026

Kernel-based External Validity of Experimental Studies

arXiv 2610.13625 ↗First posted 9 Oct 2026stat.ME cs.LG stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

The external validity of a study is the extent to which its findings generalize to unseen experimental conditions, such as datasets. We formalize the results of a study together with its research question, and define external validity as the probability that two independent and same-sized replications of the study produce results whose maximum mean discrepancy (MMD) is below a given tolerance. This approach, centered on the choice of a space of results and of an appropriate reproducing kernel for the MMD, is flexible enough to be deployed on multiple different types of results and research questions. Additionally, we establish an asymptotic power-law relation between the size of a study and the quantiles of the MMD, which we use to estimate the desired size of an externally valid study from a small-scale preliminary study. We deploy our framework to rankings with ties and discuss kernels suited to different research questions. Experiments on existing case studies and on a simulated newly designed study illustrate the applicability of the framework. We release our code as part of the genexpy Python module.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 44 of its 90 references that are in the map. First posted on arXiv 9 Oct 2026.

Builds on

44 of 90 references.

References · 44

  1. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  2. 2020Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)cited by 322
  3. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningcited by 263
  4. 2015Generative Moment Matching Networkscited by 257
  5. 2017MMD GAN: Towards Deeper Understanding of Moment Matching Networkcited by 254
  6. 2025LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristicscited by 254
  7. 2021Accounting for Variance in Machine Learning Benchmarkscited by 201
  8. 2015Training generative neural networks via Maximum Mean Discrepancy optimizationcited by 151
  9. 2024When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboardscited by 137
  10. 2021The Benchmark Lotterycited by 136
  11. 2020Learning Deep Kernels for Non-Parametric Two-Sample Testscited by 130
  12. 2019Maximum Mean Discrepancy Gradient Flowcited by 127
32 more
  1. 2025Inductive Moment Matchingcited by 113
  2. 2024When combinations of humans and AI are useful: A systematic review and meta-analysiscited by 106
  3. 2016Generative Models and Model Criticism via Optimized Maximum Mean Discrepancycited by 100
  4. 2016Time for a change: a tutorial for comparing multiple classifiers through Bayesian analysiscited by 93
  5. 2021Randomness In Neural Network Training: Characterizing The Impact of Toolingcited by 59
  6. 2018Signature moments to characterize laws of stochastic processescited by 56
  7. 2021Evaluation Metrics for Graph Generative Models: Problems, Pitfalls, and Practical Solutionscited by 39
  8. 2022Sources of Irreproducibility in Machine Learning: A Reviewcited by 32
  9. 2014A Wild Bootstrap for Degenerate Kernel Testscited by 29
  10. 2015Scan B-Statistic for Kernel Change-Point Detectioncited by 29
  11. 2020A Kernel Two-Sample Test for Functional Datacited by 28
  12. 2024Inherent Trade-Offs between Diversity and Stability in Multi-Task Benchmarkscited by 28
  13. 2020Research Reproducibility as a Survival Analysiscited by 16
  14. 2016Statistical comparison of classifiers through Bayesian hierarchical modellingcited by 14
  15. 2025Weighted quantization using MMD: From mean field to mean shift via gradient flowscited by 12
  16. 2018Antithetic and Monte Carlo kernel estimators for partial rankingscited by 11
  17. 2022A Revealing Large-Scale Evaluation of Unsupervised Anomaly Detection Algorithmscited by 11
  18. 2018A maximum-mean-discrepancy goodness-of-fit test for censored datacited by 9
  19. 2024Interaction-Force Transport Gradient Flowscited by 9
  20. 2016On kernel methods for covariates that are rankingscited by 8
  21. 2021Kernel Two-Sample Tests for Manifold Datacited by 8
  22. 2022Does the Market of Citations Reward Reproducible Work?cited by 8
  23. 2023RD-Suite: A Benchmark for Ranking Distillationcited by 8
  24. 2025KerJEPA: Kernel Discrepancies for Euclidean Self-Supervised Learningcited by 8
  25. 2022Maximum Mean Discrepancy on Exponential Windows for Online Change Detectioncited by 6
  26. 2023A benchmark of categorical encoders for binary classificationcited by 6
  27. 2025A Scalable Nystrom-Based Kernel Two-Sample Test with Permutationscited by 6
  28. 2020Kernel Two-Sample and Independence Tests for Non-Stationary Random Processescited by 5
  29. 2023A Coreset Learning Reality Checkcited by 5
  30. 2022On the Generalizability and Predictability of Recommender Systemscited by 4
  31. 2025Adversarial Subspace Generation for Outlier Detection in High-Dimensional Datacited by 2
  32. 2025Optimal Online Change Detection via Random Fourier Featurescited by 2

Built on by

Not cited yet.