# A label-efficient two-sample test

arXiv: https://arxiv.org/abs/2111.08861 · HTML: https://mlsanity.com/paper/2111.08861/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 33 references that are in the map.
- First posted on arXiv 17 Nov 2021.

## Builds on (most cited first, 6 of 6)

- [Revisiting Classifier Two-Sample Tests](https://mlsanity.com/paper/1610.06545/) (2016)
- [Active Testing: Sample-Efficient Model Evaluation](https://mlsanity.com/paper/2103.05331/) (2021)
- [A new graph-based two-sample test for multivariate and object data](https://mlsanity.com/paper/1307.6294/) (2013)
- [S2: An Efficient Graph Based Active Learning Algorithm with Application to Nonparametric Classification](https://mlsanity.com/paper/1506.08760/) (2015)
- [Finding the Homology of Decision Boundaries with Active Learning](https://mlsanity.com/paper/2011.09645/) (2020)
- [A weighted edge-count two-sample test for multivariate and object data](https://mlsanity.com/paper/1604.06515/) (2016)

## Built on by (most cited first, 4 of 4)

- [Statistically Valid Post-Deployment Monitoring Should Be Standard for AI-Based Digital Health](https://mlsanity.com/paper/2506.05701/) (2025)
- [Active Sequential Two-Sample Testing](https://mlsanity.com/paper/2301.12616/) (2023)
- [Advanced Tutorial: Label-Efficient Two-Sample Tests](https://mlsanity.com/paper/2501.03568/) (2025)
- [Matched-Pair Experimental Design with Active Learning](https://mlsanity.com/paper/2509.10742/) (2025)

## Abstract (arXiv)

Two-sample tests evaluate whether two samples are realizations of the same distribution (the null hypothesis) or two different distributions (the alternative hypothesis). We consider a new setting for this problem where sample features are easily measured whereas sample labels are unknown and costly to obtain. Accordingly, we devise a three-stage framework in service of performing an effective two-sample test with only a small number of sample label queries: first, a classifier is trained with samples uniformly labeled to model the posterior probabilities of the labels; second, a novel query scheme dubbed bimodal query is used to query labels of samples from both classes, and last, the classical Friedman-Rafsky (FR) two-sample test is performed on the queried samples. Theoretical analysis and extensive experiments performed on several datasets demonstrate that the proposed test controls the Type I error and has decreased Type II error relative to uniform querying and certainty-based querying. Source code for our algorithms and experimental results is available at urlhttps://github.com/wayne0908/Label-Efficient-Two-Sample.