# Even Small Correlation and Diversity Shifts Pose Dataset-Bias Issues

arXiv: https://arxiv.org/abs/2305.05807 · HTML: https://mlsanity.com/paper/2305.05807/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 41 references that are in the map.
- First posted on arXiv 9 May 2023.

## Builds on (most cited first, 25 of 25)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization](https://mlsanity.com/paper/1911.08731/) (2019)
- [Deep Hashing Network for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1706.07522/) (2017)
- [Deeper, Broader and Artier Domain Generalization](https://mlsanity.com/paper/1710.03077/) (2017)
- [Adversarial Examples Are Not Bugs, They Are Features](https://mlsanity.com/paper/1905.02175/) (2019)
- [Towards Stable Test-Time Adaptation in Dynamic Wild World](https://mlsanity.com/paper/2302.12400/) (2023)
- [Environment Inference for Invariant Learning](https://mlsanity.com/paper/2010.07249/) (2020)
- [Gradient Starvation: A Learning Proclivity in Neural Networks](https://mlsanity.com/paper/2011.09468/) (2020)
- [BCN20000: Dermoscopic Lesions in the Wild](https://mlsanity.com/paper/1908.02288/) (2019)
- [Reading Race: AI Recognises Patient's Racial Identity In Medical Images](https://mlsanity.com/paper/2107.10356/) (2021)
- [An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Models](https://mlsanity.com/paper/2007.06778/) (2020)
- [PAD-UFES-20: a skin lesion dataset composed of patient data and clinical images collected from smartphones](https://mlsanity.com/paper/2007.00478/) (2020)
- [The iWildCam 2020 Competition Dataset](https://mlsanity.com/paper/2004.10340/) (2020)
- [Can Subnetwork Structure be the Key to Out-of-Distribution Generalization?](https://mlsanity.com/paper/2106.02890/) (2021)
- [Debiasing Skin Lesion Datasets and Models? Not So Fast](https://mlsanity.com/paper/2004.11457/) (2020)
- [(De)Constructing Bias on Skin Lesion Datasets](https://mlsanity.com/paper/1904.08818/) (2019)
- [Artifact-Based Domain Generalization of Skin Lesion Models](https://mlsanity.com/paper/2208.09756/) (2022)
- [Domain shifts in dermoscopic skin cancer datasets: Evaluation of essential limitations for clinical translation](https://mlsanity.com/paper/2304.06968/) (2023)
- [Non-Verbal Communication Analysis in Victim-Offender Mediations](https://mlsanity.com/paper/1412.2122/) (2014)

## Built on by (most cited first, 8 of 8)

- [BiasPruner: Debiased Continual Learning for Medical Image Classification](https://mlsanity.com/paper/2407.08609/) (2024)
- [Key Patches Are All You Need: A Multiple Instance Learning Framework For Robust Medical Diagnosis](https://mlsanity.com/paper/2405.01654/) (2024)
- [Subgroup Performance Analysis in Hidden Stratifications](https://mlsanity.com/paper/2503.10382/) (2025)
- [UnLearning from Experience to Avoid Spurious Correlations](https://mlsanity.com/paper/2409.02792/) (2024)
- [Test-Time Selection for Robust Skin Lesion Analysis](https://mlsanity.com/paper/2308.05595/) (2023)
- [A New Perspective On AI Safety Through Control Theory Methodologies](https://mlsanity.com/paper/2506.23703/) (2025)
- [Are Explanations Helpful? A Comparative Analysis of Explainability Methods in Skin Lesion Classifiers](https://mlsanity.com/paper/2412.03166/) (2024)
- [A Concept for Efficient Scalability of Automated Driving Allowing for Technical, Legal, Cultural, and Ethical Differences](https://mlsanity.com/paper/2507.18326/) (2025)

## Abstract (arXiv)

Distribution shifts are common in real-world datasets and can affect the performance and reliability of deep learning models. In this paper, we study two types of distribution shifts: diversity shifts, which occur when test samples exhibit patterns unseen during training, and correlation shifts, which occur when test data present a different correlation between seen invariant and spurious features. We propose an integrated protocol to analyze both types of shifts using datasets where they co-exist in a controllable manner. Finally, we apply our approach to a real-world classification problem of skin cancer analysis, using out-of-distribution datasets and specialized bias annotations. Our protocol reveals three findings: 1) Models learn and propagate correlation shifts even with low-bias training; this poses a risk of accumulating and combining unaccountable weak biases; 2) Models learn robust features in high- and low-bias scenarios but use spurious ones if test samples have them; this suggests that spurious correlations do not impair the learning of robust features; 3) Diversity shift can reduce the reliance on spurious correlations; this is counter intuitive since we expect biased models to depend more on biases when invariant features are missing. Our work has implications for distribution shift research and practice, providing new insights into how models learn and rely on spurious correlations under different types of shifts.