# A Kernel-Based Conditional Two-Sample Test Using Nearest Neighbors (with Applications to Calibration, Regression Curves, and Simulation-Based Inference)

arXiv: https://arxiv.org/abs/2407.16550 · HTML: https://mlsanity.com/paper/2407.16550/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 132 references that are in the map.
- First posted on arXiv 23 Jul 2024.

## Builds on (most cited first, 30 of 48)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Normalizing Flows for Probabilistic Modeling and Inference](https://mlsanity.com/paper/1912.02762/) (2019)
- [Revisiting Classifier Two-Sample Tests](https://mlsanity.com/paper/1610.06545/) (2016)
- [Sequential Neural Likelihood: Fast Likelihood-free Inference with Autoregressive Flows](https://mlsanity.com/paper/1805.07226/) (2018)
- [Automatic Posterior Transformation for Likelihood-Free Inference](https://mlsanity.com/paper/1905.07488/) (2019)
- [Conditional Image Generation with Score-Based Diffusion Models](https://mlsanity.com/paper/2111.13606/) (2021)
- [Evaluating model calibration in classification](https://mlsanity.com/paper/1902.06977/) (2019)
- [Benchmarking Simulation-Based Inference](https://mlsanity.com/paper/2101.04653/) (2021)
- [Validating Bayesian Inference Algorithms with Simulation-Based Calibration](https://mlsanity.com/paper/1804.06788/) (2018)
- [Flow Matching for Scalable Simulation-Based Inference](https://mlsanity.com/paper/2305.17161/) (2023)
- [On Contrastive Learning for Likelihood-free Inference](https://mlsanity.com/paper/2002.03712/) (2020)
- [A Measure-Theoretic Approach to Kernel Conditional Mean Embeddings](https://mlsanity.com/paper/2002.03689/) (2020)
- [MMD Aggregated Two-Sample Test](https://mlsanity.com/paper/2110.15073/) (2021)
- [All-in-one simulation-based inference](https://mlsanity.com/paper/2404.09636/) (2024)
- [Conditional Simulation Using Diffusion Schr"odinger Bridges](https://mlsanity.com/paper/2202.13460/) (2022)
- [Conditional Density Estimation with Bayesian Normalising Flows](https://mlsanity.com/paper/1802.04908/) (2018)
- [Sampling-Based Accuracy Testing of Posterior Estimators for General Inference](https://mlsanity.com/paper/2302.03026/) (2023)
- [Variational methods for simulation-based inference](https://mlsanity.com/paper/2203.04176/) (2022)
- [Truncated proposals for scalable and hassle-free simulation-based inference](https://mlsanity.com/paper/2210.04815/) (2022)
- [GATSBI: Generative Adversarial Training for Simulation-Based Inference](https://mlsanity.com/paper/2203.06481/) (2022)
- [Autoregressive Conditional Neural Processes](https://mlsanity.com/paper/2303.14468/) (2023)
- [Metrics of calibration for probabilistic predictions](https://mlsanity.com/paper/2205.09680/) (2022)
- [Sequential Neural Posterior and Likelihood Approximation](https://mlsanity.com/paper/2102.06522/) (2021)
- [Diagnostics for Conditional Density Models and Bayesian Inference Algorithms](https://mlsanity.com/paper/2102.10473/) (2021)
- [Conditional Generative Moment-Matching Networks](https://mlsanity.com/paper/1606.04218/) (2016)
- [Wasserstein Generative Learning of Conditional Distribution](https://mlsanity.com/paper/2112.10039/) (2021)
- [T-Cal: An optimal test for the calibration of predictive models](https://mlsanity.com/paper/2203.01850/) (2022)
- [Calibration tests beyond classification](https://mlsanity.com/paper/2210.13355/) (2022)
- [HNPE: Leveraging Global Parameters for Neural Posterior Estimation](https://mlsanity.com/paper/2102.06477/) (2021)

## Built on by (most cited first, 7 of 7)

- [General Frameworks for Conditional Two-Sample Testing](https://mlsanity.com/paper/2410.16636/) (2024)
- [All Models Are Miscalibrated, But Some Less So: Comparing Calibration with Conditional Mean Operators](https://mlsanity.com/paper/2502.11465/) (2025)
- [Prompting Particle Physics: Tokenized Multi-modal Foundation Models for Combinatorially Many Tasks](https://mlsanity.com/paper/2609.31862/) (2026)
- [Measuring Differences between Conditional Distributions using Kernel Embeddings](https://mlsanity.com/paper/2605.02260/) (2026)
- [Kernel conditional tests from learning-theoretic bounds](https://mlsanity.com/paper/2506.03898/) (2025)
- [One-shot Conditional Sampling: MMD meets Nearest Neighbors](https://mlsanity.com/paper/2509.25507/) (2025)
- [Testing Conditional Distribution Equality Using Generative Models](https://mlsanity.com/paper/2610.20447/) (2026)

## Abstract (arXiv)

In this paper we introduce a kernel-based measure for detecting differences between two conditional distributions. Using the `kernel trick' and nearest-neighbor graphs, we propose a consistent estimate of this measure which can be computed in nearly linear time (for a fixed number of nearest neighbors). Moreover, when the two conditional distributions are the same, the estimate has a Gaussian limit and its asymptotic variance has a simple form that can be easily estimated from the data. The resulting test attains precise asymptotic level and is universally consistent for detecting differences between two conditional distributions. We also provide a resampling based test using our estimate that applies to the conditional goodness-of-fit problem, which controls Type I error in finite samples and is asymptotically consistent with only a finite number of resamples. A method to de-randomize the resampling test is also presented. The proposed methods can be readily applied to a broad range of problems, ranging from classical nonparametric statistics to modern machine learning. Specifically, we explore three applications: testing model calibration, regression curve evaluation, and validation of emulator models in simulation-based inference. We illustrate the superior performance of our method for these tasks, both in simulations as well as on real data. In particular, we apply our method to (1) assess the calibration of neural network models trained on the CIFAR-10 dataset, (2) compare regression functions for wind power generation across two different turbines, and (3) validate emulator models on benchmark examples with intractable posteriors and for generating synthetic `redshift' associated with galaxy images.