# Statistical quantification of confounding bias in predictive modelling

arXiv: https://arxiv.org/abs/2111.00814 · HTML: https://mlsanity.com/paper/2111.00814/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 73 references that are in the map.
- First posted on arXiv 1 Nov 2021.

## Builds on (most cited first, 6 of 6)

- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [Causal inference using invariant prediction: identification and confidence intervals](https://mlsanity.com/paper/1501.01332/) (2015)
- [Panning for Gold: Model-X Knockoffs for High-dimensional Controlled Variable Selection](https://mlsanity.com/paper/1610.02351/) (2016)
- [Detect and Correct Bias in Multi-Site Neuroimaging Datasets](https://mlsanity.com/paper/2002.05049/) (2020)
- [Preventing dataset shift from breaking machine-learning biomarkers](https://mlsanity.com/paper/2107.09947/) (2021)
- [Measuring the effects of confounders in medical supervised classification problems: the Confounding Index (CI)](https://mlsanity.com/paper/1905.08871/) (2019)

## Built on by (most cited first, 7 of 7)

- [Deep Nonparametric Conditional Independence Tests for Images](https://mlsanity.com/paper/2411.06140/) (2024)
- [A survey on bias in machine learning research](https://mlsanity.com/paper/2308.11254/) (2023)
- [DeepRepViz: Identifying Confounders in Deep Learning Model Predictions](https://mlsanity.com/paper/2309.15551/) (2023)
- [Confound-leakage: Confound Removal in Machine Learning Leads to Leakage](https://mlsanity.com/paper/2210.09232/) (2022)
- [Common Steps in Machine Learning Might Hinder The Explainability Aims in Medicine](https://mlsanity.com/paper/2409.00155/) (2024)
- [ICON decomposition: Multivariate concept auditing reveals shortcuts that neural networks rely on](https://mlsanity.com/paper/2608.26083/) (2026)
- [Event Cameras for Melt-Pool Monitoring in Additive Manufacturing: A Benchmark and a Cross-Machine Transfer Analysis](https://mlsanity.com/paper/2610.06973/) (2026)

## Abstract (arXiv)

The lack of non-parametric statistical tests for confounding bias significantly hampers the development of robust, valid and generalizable predictive models in many fields of research. Here I propose the partial and full confounder tests, which, for a given confounder variable, probe the null hypotheses of unconfounded and fully confounded models, respectively. The tests provide a strict control for Type I errors and high statistical power, even for non-normally and non-linearly dependent predictions, often seen in machine learning. Applying the proposed tests on models trained on functional brain connectivity data from the Human Connectome Project and the Autism Brain Imaging Data Exchange dataset reveals confounders that were previously unreported or found to be hard to correct for with state-of-the-art confound mitigation approaches. The tests, implemented in the package mlconfound (https://mlconfound.readthedocs.io), can aid the assessment and improvement of the generalizability and neurobiological validity of predictive models and, thereby, foster the development of clinically useful machine learning biomarkers.