# Conformal Data Contamination Tests for In-distribution Data Acquisition

arXiv: https://arxiv.org/abs/2507.13835 · HTML: https://mlsanity.com/paper/2507.13835/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 66 references that are in the map.
- First posted on arXiv 18 Jul 2025.

## Builds on (most cited first, 30 of 31)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks](https://mlsanity.com/paper/1807.03888/) (2018)
- [LEAF: A Benchmark for Federated Settings](https://mlsanity.com/paper/1812.01097/) (2018)
- [Data Shapley: Equitable Valuation of Data for Machine Learning](https://mlsanity.com/paper/1904.02868/) (2019)
- [Conformal Prediction Under Covariate Shift](https://mlsanity.com/paper/1904.06019/) (2019)
- [Conformal prediction beyond exchangeability](https://mlsanity.com/paper/2202.13415/) (2022)
- [Small Language Models are the Future of Agentic AI](https://mlsanity.com/paper/2506.02153/) (2025)
- [Model Context Protocol (MCP): Landscape, Security Threats, and Future Research Directions](https://mlsanity.com/paper/2503.23278/) (2025)
- [Testing for Outliers with Conformal p-values](https://mlsanity.com/paper/2104.08279/) (2021)
- [Data Valuation using Reinforcement Learning](https://mlsanity.com/paper/1909.11671/) (2019)
- [Distribution-free uncertainty quantification for classification under label shift](https://mlsanity.com/paper/2103.03323/) (2021)
- [Mixture Proportion Estimation via Kernel Embedding of Distributions](https://mlsanity.com/paper/1603.02501/) (2016)
- [Transductive conformal inference with adaptive scores](https://mlsanity.com/paper/2310.18108/) (2023)
- [Detecting Adversarial Data by Probing Multiple Perturbations Using Expected Perturbation Score](https://mlsanity.com/paper/2305.16035/) (2023)
- [Adaptive novelty detection with false discovery rate guarantee](https://mlsanity.com/paper/2208.06685/) (2022)
- [Estimating the Contamination Factor's Distribution in Unsupervised Anomaly Detection](https://mlsanity.com/paper/2210.10487/) (2022)
- [Robust Conformal Outlier Detection under Contaminated Reference Data](https://mlsanity.com/paper/2502.04807/) (2025)
- [Multi-Scale Convolutional Neural Network for Automated AMD Classification using Retinal OCT Images](https://mlsanity.com/paper/2110.03002/) (2021)
- [Mixture Proportion Estimation Beyond Irreducibility](https://mlsanity.com/paper/2306.01253/) (2023)
- [Evaluating and Incentivizing Diverse Data Contributions in Collaborative Learning](https://mlsanity.com/paper/2306.05592/) (2023)
- [Delegating Data Collection in Decentralized Machine Learning](https://mlsanity.com/paper/2309.01837/) (2023)
- [Semi-supervised multiple testing](https://mlsanity.com/paper/2106.13501/) (2021)
- [Robust Conformal Prediction Using Privileged Information](https://mlsanity.com/paper/2406.05405/) (2024)
- [DAVED: Data Acquisition via Experimental Design for Data Markets](https://mlsanity.com/paper/2403.13893/) (2024)
- [Conformal Prediction Under Generalized Covariate Shift with Posterior Drift](https://mlsanity.com/paper/2502.17744/) (2025)
- [The power of visualizing distributional differences: Formal graphical n-sample tests](https://mlsanity.com/paper/2403.01838/) (2024)

## Built on by (most cited first, 1 of 1)

- [Online Continual Learning for Anomaly Detection in IoT under Data Distribution Shifts](https://mlsanity.com/paper/2603.07507/) (2026)

## Abstract (arXiv)

The amount of quality data in many machine learning tasks is limited to what is available locally to data owners. The set of quality data can be expanded through trading or sharing with external data agents. However, external data may be contaminated or introduce undesirable sample diversity which can degrade performance of personalized machine learning tasks, as in diagnosis of a rare disease or recommendation systems. Therefore, data buyers need quality guarantees prior to data acquisition. Previous works primarily rely on distributional assumptions about data from different agents, relegating quality checks to post-hoc steps involving costly data valuation procedures. We propose a distribution-free, contamination-aware data acquisition framework that, by inspecting only a small volume of data, identifies external data agents whose data is most valuable for model personalization. To achieve this, we introduce novel two-sample testing procedures, preceding full data acquisition, grounded in rigorous theoretical foundations for conformal outlier detection, to determine whether an agent's data exceeds a contamination threshold. The proposed tests, termed conformal data contamination tests, remain valid under arbitrary contamination levels and the novel Storey-type test provably enables finite-sample false discovery rate control via the Benjamini-Hochberg procedure. Empirical evaluations across diverse collaborative learning scenarios demonstrate the robustness and effectiveness of our approach. Overall, the conformal data contamination test distinguishes itself as a generic procedure for aggregating data with statistically rigorous quality guarantees.