# A Two-Sample Conditional Distribution Test Using Conformal Prediction and Weighted Rank Sum

arXiv: https://arxiv.org/abs/2010.07147 · HTML: https://mlsanity.com/paper/2010.07147/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Oct 2020.

## Built on by (most cited first, 19 of 19)

- [Online multiple testing with e-values](https://mlsanity.com/paper/2311.06412/) (2023)
- [Conformal link prediction for false discovery rate control](https://mlsanity.com/paper/2306.14693/) (2023)
- [Distance and Kernel-Based Measures for Global and Local Two-Sample Conditional Distribution Testing](https://mlsanity.com/paper/2210.08149/) (2022)
- [Efficient and Multiply Robust Risk Estimation under General Forms of Dataset Shift](https://mlsanity.com/paper/2306.16406/) (2023)
- [Theoretical guarantees for change localization using conformal p-values](https://mlsanity.com/paper/2510.08749/) (2025)
- [QQWorld: Quantile-Quantile Matching for World Model Regularization](https://mlsanity.com/paper/2607.28415/) (2026)
- [Uncertainty quantification in metric spaces](https://mlsanity.com/paper/2405.05110/) (2024)
- [Optimal Aggregation of Prediction Intervals under Unsupervised Domain Shift](https://mlsanity.com/paper/2405.10302/) (2024)
- [General Frameworks for Conditional Two-Sample Testing](https://mlsanity.com/paper/2410.16636/) (2024)
- [A note on conditional PAC-efficient reasoning in large language model routing](https://mlsanity.com/paper/2512.03057/) (2025)
- [Conformal and kNN Predictive Uncertainty Quantification Algorithms in Metric Spaces](https://mlsanity.com/paper/2507.15741/) (2025)
- [Conformal C2ST: Turning weak classifiers into strong two-sample tests](https://mlsanity.com/paper/2507.17026/) (2025)
- [A Conditional Distribution Equality Testing Framework using Deep Generative Learning](https://mlsanity.com/paper/2509.17729/) (2025)
- [ConfHit: Conformal Generative Design with Oracle Free Guarantees](https://mlsanity.com/paper/2603.07371/) (2026)
- [Separating Covariate Shift from Mechanism Change with Two Discriminators: CJSD, a Conditional Discrepancy with an Exact Covariate-Concept Decomposition](https://mlsanity.com/paper/2608.19885/) (2026)
- [Assumption-free fidelity bounds for hardware noise characterization](https://mlsanity.com/paper/2504.07010/) (2025)
- [Kernel conditional tests from learning-theoretic bounds](https://mlsanity.com/paper/2506.03898/) (2025)
- [Machine-Learning-Assisted Comparison of Regression Functions](https://mlsanity.com/paper/2510.24714/) (2025)
- [Testing Conditional Distribution Equality Using Generative Models](https://mlsanity.com/paper/2610.20447/) (2026)

## Abstract (arXiv)

We consider the problem of testing the equality of conditional distributions of a response variable given a vector of covariates between two populations. Such a hypothesis testing problem can be motivated from various machine learning and statistical inference scenarios, including transfer learning and causal predictive inference. We develop a nonparametric test procedure inspired from the conformal prediction framework. The construction of our test statistic combines recent developments in conformal prediction with a novel choice of conformity score, resulting in a weighted rank-sum test statistic that is valid and powerful under general settings. To our knowledge, this is the first successful attempt of using conformal prediction for testing statistical hypotheses beyond exchangeability. Our method is suitable for modern machine learning scenarios where the data has high dimensionality and large sample sizes, and can be effectively combined with existing classification algorithms to find good conformity score functions. The performance of the proposed method is demonstrated in various numerical examples.