# Pulling Up by the Causal Bootstraps: Causal Data Augmentation for Pre-training Debiasing

arXiv: https://arxiv.org/abs/2108.12510 · HTML: https://mlsanity.com/paper/2108.12510/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 61 references that are in the map.
- First posted on arXiv 27 Aug 2021.

## Builds on (most cited first, 30 of 31)

- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [End to End Learning for Self-Driving Cars](https://mlsanity.com/paper/1604.07316/) (2016)
- [Recognition in Terra Incognita](https://mlsanity.com/paper/1807.04975/) (2018)
- [Causal inference using invariant prediction: identification and confidence intervals](https://mlsanity.com/paper/1501.01332/) (2015)
- [Confounding variables can degrade generalization performance of radiological deep learning models](https://mlsanity.com/paper/1807.00431/) (2018)
- [Learning Not to Learn: Training Deep Neural Networks with Biased Data](https://mlsanity.com/paper/1812.10352/) (2018)
- [What is the Effect of Importance Weighting in Deep Learning?](https://mlsanity.com/paper/1812.03372/) (2018)
- [Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift](https://mlsanity.com/paper/1810.11953/) (2018)
- [Gender Bias in Contextualized Word Embeddings](https://mlsanity.com/paper/1904.03310/) (2019)
- [Learning De-biased Representations with Biased Representations](https://mlsanity.com/paper/1910.02806/) (2019)
- [Invariant Causal Prediction for Nonlinear Models](https://mlsanity.com/paper/1706.08576/) (2017)
- [Ethical Machine Learning in Health Care](https://mlsanity.com/paper/2009.10576/) (2020)
- [Cycle-Consistency for Robust Visual Question Answering](https://mlsanity.com/paper/1902.05660/) (2019)
- [Measuring the tendency of CNNs to Learn Surface Statistical Regularities](https://mlsanity.com/paper/1711.11561/) (2017)
- [Model Patching: Closing the Subgroup Performance Gap with Data Augmentation](https://mlsanity.com/paper/2008.06775/) (2020)
- [Hurtful Words: Quantifying Biases in Clinical Contextual Word Embeddings](https://mlsanity.com/paper/2003.11515/) (2020)
- [A Causal View on Robustness of Neural Networks](https://mlsanity.com/paper/2005.01095/) (2020)
- [Contemplating Visual Emotions: Understanding and Overcoming Dataset Bias](https://mlsanity.com/paper/1808.02212/) (2018)
- [Does Distributionally Robust Supervised Learning Give Robust Classifiers?](https://mlsanity.com/paper/1611.02041/) (2016)
- [Weakly-supervised learning of visual relations](https://mlsanity.com/paper/1707.09472/) (2017)
- [Clinical Intervention Prediction and Understanding using Deep Networks](https://mlsanity.com/paper/1705.08498/) (2017)
- [Towards Accuracy-Fairness Paradox: Adversarial Example-based Data Augmentation for Visual Debiasing](https://mlsanity.com/paper/2007.13632/) (2020)
- [Causal bootstrapping](https://mlsanity.com/paper/1910.09648/) (2019)
- [An Adversarial Approach for the Robust Classification of Pneumonia from Chest Radiographs](https://mlsanity.com/paper/2001.04051/) (2020)
- [A Causal Bootstrap](https://mlsanity.com/paper/1807.02737/) (2018)
- [I-SPEC: An End-to-End Framework for Learning Transportable, Shift-Stable Models](https://mlsanity.com/paper/2002.08948/) (2020)

## Built on by (most cited first, 6 of 6)

- [Change is Hard: A Closer Look at Subpopulation Shift](https://mlsanity.com/paper/2302.12254/) (2023)
- [The Road to Explainability is Paved with Bias: Measuring the Fairness of Explanations](https://mlsanity.com/paper/2205.03295/) (2022)
- [OpinionGPT: Modelling Explicit Biases in Instruction-Tuned LLMs](https://mlsanity.com/paper/2309.03876/) (2023)
- [Not Just Pretty Pictures: Toward Interventional Data Augmentation Using Text-to-Image Generators](https://mlsanity.com/paper/2212.11237/) (2022)
- [Stubborn Lexical Bias in Data and Models](https://mlsanity.com/paper/2306.02190/) (2023)
- [Undersmoothing Causal Estimators with Generative Trees](https://mlsanity.com/paper/2203.08570/) (2022)

## Abstract (arXiv)

Machine learning models achieve state-of-the-art performance on many supervised learning tasks. However, prior evidence suggests that these models may learn to rely on shortcut biases or spurious correlations (intuitively, correlations that do not hold in the test as they hold in train) for good predictive performance. Such models cannot be trusted in deployment environments to provide accurate predictions. While viewing the problem from a causal lens is known to be useful, the seamless integration of causation techniques into machine learning pipelines remains cumbersome and expensive. In this work, we study and extend a causal pre-training debiasing technique called causal bootstrapping (CB) under five practical confounded-data generation-acquisition scenarios (with known and unknown confounding). Under these settings, we systematically investigate the effect of confounding bias on deep learning model performance, demonstrating their propensity to rely on shortcut biases when these biases are not properly accounted for. We demonstrate that such a causal pre-training technique can significantly outperform existing base practices to mitigate confounding bias on real-world domain generalization benchmarking tasks. This systematic investigation underlines the importance of accounting for the underlying data-generating mechanisms and fortifying data-preprocessing pipelines with a causal framework to develop methods robust to confounding biases.