# General Frameworks for Conditional Two-Sample Testing

arXiv: https://arxiv.org/abs/2410.16636 · HTML: https://mlsanity.com/paper/2410.16636/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 65 references that are in the map.
- First posted on arXiv 22 Oct 2024.

## Builds on (most cited first, 27 of 27)

- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [Revisiting Classifier Two-Sample Tests](https://mlsanity.com/paper/1610.06545/) (2016)
- [Learning Deep Kernels for Non-Parametric Two-Sample Tests](https://mlsanity.com/paper/2002.09116/) (2020)
- [Panning for Gold: Model-X Knockoffs for High-dimensional Controlled Variable Selection](https://mlsanity.com/paper/1610.02351/) (2016)
- [Telescoping Density-Ratio Estimation](https://mlsanity.com/paper/2006.12204/) (2020)
- [Approximate Kernel-based Conditional Independence Tests for Fast Non-Parametric Causal Discovery](https://mlsanity.com/paper/1702.03877/) (2017)
- [MMD Aggregated Two-Sample Test](https://mlsanity.com/paper/2110.15073/) (2021)
- [Density Ratio Estimation via Infinitesimal Classification](https://mlsanity.com/paper/2111.11010/) (2021)
- [A general framework for inference on algorithm-agnostic variable importance](https://mlsanity.com/paper/2004.03683/) (2020)
- [Classification accuracy as a proxy for two sample testing](https://mlsanity.com/paper/1602.02210/) (2016)
- [Featurized Density Ratio Estimation](https://mlsanity.com/paper/2107.02212/) (2021)
- [Local permutation tests for conditional independence](https://mlsanity.com/paper/2112.11666/) (2021)
- [Fast and Powerful Conditional Randomization Testing via Distillation](https://mlsanity.com/paper/2006.03980/) (2020)
- [A Two-Sample Conditional Distribution Test Using Conformal Prediction and Weighted Rank Sum](https://mlsanity.com/paper/2010.07147/) (2020)
- [The Projected Covariance Measure for assumption-lean variable significance testing](https://mlsanity.com/paper/2211.02039/) (2022)
- [Trimmed Density Ratio Estimation](https://mlsanity.com/paper/1703.03216/) (2017)
- [Significance tests of feature relevance for a black-box learner](https://mlsanity.com/paper/2103.04985/) (2021)
- [Practical Kernel Tests of Conditional Independence](https://mlsanity.com/paper/2402.13196/) (2024)
- [A Short Survey on Importance Weighting for Machine Learning](https://mlsanity.com/paper/2403.10175/) (2024)
- [On the Use of Random Forest for Two-Sample Testing](https://mlsanity.com/paper/1903.06287/) (2019)
- [The Weighted Generalised Covariance Measure](https://mlsanity.com/paper/2111.04361/) (2021)
- [Global and Local Two-Sample Tests via Regression](https://mlsanity.com/paper/1812.08927/) (2018)
- [Distance and Kernel-Based Measures for Global and Local Two-Sample Conditional Distribution Testing](https://mlsanity.com/paper/2210.08149/) (2022)
- [A Kernel-Based Conditional Two-Sample Test Using Nearest Neighbors (with Applications to Calibration, Regression Curves, and Simulation-Based Inference)](https://mlsanity.com/paper/2407.16550/) (2024)
- [De-Biased Two-Sample U-Statistics With Application To Conditional Distribution Testing](https://mlsanity.com/paper/2402.00164/) (2024)
- [Conditional Independence Testing for Discrete Distributions: Beyond chi2- and G-tests](https://mlsanity.com/paper/2308.05373/) (2023)
- [Doubly robust and computationally efficient high-dimensional variable selection](https://mlsanity.com/paper/2409.09512/) (2024)

## Built on by (most cited first, 4 of 4)

- [Prompting Particle Physics: Tokenized Multi-modal Foundation Models for Combinatorially Many Tasks](https://mlsanity.com/paper/2609.31862/) (2026)
- [Measuring Differences between Conditional Distributions using Kernel Embeddings](https://mlsanity.com/paper/2605.02260/) (2026)
- [Separating Covariate Shift from Mechanism Change with Two Discriminators: CJSD, a Conditional Discrepancy with an Exact Covariate-Concept Decomposition](https://mlsanity.com/paper/2608.19885/) (2026)
- [Kernel conditional tests from learning-theoretic bounds](https://mlsanity.com/paper/2506.03898/) (2025)

## Abstract (arXiv)

We study the problem of conditional two-sample testing, which aims to determine whether two populations have the same distribution after accounting for confounding factors. This problem commonly arises in various applications, such as domain adaptation and algorithmic fairness, where comparing two groups is essential while controlling for confounding variables. We begin by establishing a hardness result for conditional two-sample testing, demonstrating that no valid test can have significant power against any single alternative without proper assumptions. We then introduce two general frameworks that implicitly or explicitly target specific classes of distributions for their validity and power. Our first framework allows us to convert any conditional independence test into a conditional two-sample test in a black-box manner, while preserving the asymptotic properties of the original conditional independence test. The second framework transforms the problem into comparing marginal distributions with estimated density ratios, which allows us to leverage existing methods for marginal two-sample testing. We demonstrate this idea in a concrete manner with classification and kernel-based methods. Finally, simulation studies are conducted to illustrate the proposed frameworks in finite-sample scenarios.