# Spawrious: A Benchmark for Fine Control of Spurious Correlation Biases

arXiv: https://arxiv.org/abs/2303.05470 · HTML: https://mlsanity.com/paper/2303.05470/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Builds on 81 of its 123 references that are in the map.
- First posted on arXiv 9 Mar 2023.

## Builds on (most cited first, 30 of 81)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022)
- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization](https://mlsanity.com/paper/1911.08731/) (2019)
- [Deep CORAL: Correlation Alignment for Deep Domain Adaptation](https://mlsanity.com/paper/1607.01719/) (2016)
- [In Search of Lost Domain Generalization](https://mlsanity.com/paper/2007.01434/) (2020)
- [Out-of-Distribution Generalization via Risk Extrapolation (REx)](https://mlsanity.com/paper/2003.00688/) (2020)
- [Extracting Training Data from Diffusion Models](https://mlsanity.com/paper/2301.13188/) (2023)
- [Domain Generalization via Invariant Feature Representation](https://mlsanity.com/paper/1301.2115/) (2013)
- [Causal inference using invariant prediction: identification and confidence intervals](https://mlsanity.com/paper/1501.01332/) (2015)
- [Just Train Twice: Improving Group Robustness without Training Group Information](https://mlsanity.com/paper/2107.09044/) (2021)
- [Underspecification Presents Challenges for Credibility in Modern Machine Learning](https://mlsanity.com/paper/2011.03395/) (2020)
- [Meta-learners for Estimating Heterogeneous Treatment Effects using Machine Learning](https://mlsanity.com/paper/1706.03461/) (2017)
- [SWAD: Domain Generalization by Seeking Flat Minima](https://mlsanity.com/paper/2102.08604/) (2021)
- [Domain Generalization via Model-Agnostic Learning of Semantic Features](https://mlsanity.com/paper/1910.13580/) (2019)
- [Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations](https://mlsanity.com/paper/2204.02937/) (2022)
- [Noise or Signal: The Role of Image Backgrounds in Object Recognition](https://mlsanity.com/paper/2006.09994/) (2020)

## Built on by (most cited first, 30 of 41)

- [Challenges and Applications of Large Language Models](https://mlsanity.com/paper/2307.10169/) (2023)
- [REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewards](https://mlsanity.com/paper/2505.24760/) (2025)
- [The MiniPile Challenge for Data-Efficient Language Models](https://mlsanity.com/paper/2304.08442/) (2023)
- [A Multimodal Automated Interpretability Agent](https://mlsanity.com/paper/2404.14394/) (2024)
- [COCO-Counterfactuals: Automatically Constructed Counterfactual Examples for Image-Text Pairs](https://mlsanity.com/paper/2309.14356/) (2023)
- [Machine Vision Therapy: Multimodal Large Language Models Can Enhance Visual Robustness via Denoising In-Context Learning](https://mlsanity.com/paper/2312.02546/) (2023)
- [Navigating Shortcuts, Spurious Correlations, and Confounders: From Origins via Detection to Mitigation](https://mlsanity.com/paper/2412.05152/) (2024)
- [A Survey on Evaluation of Out-of-Distribution Generalization](https://mlsanity.com/paper/2403.01874/) (2024)
- [Rethinking the Evaluation Protocol of Domain Generalization](https://mlsanity.com/paper/2305.15253/) (2023)
- [TTIDA: Controllable Generative Data Augmentation via Text-to-Text and Text-to-Image Models](https://mlsanity.com/paper/2304.08821/) (2023)
- [Escaping the SpuriVerse: Can Large Vision-Language Models Generalize Beyond Seen Spurious Correlations?](https://mlsanity.com/paper/2506.18322/) (2025)
- [Navigating the Shortcut Maze: A Comprehensive Analysis of Shortcut Learning in Text Classification by Language Models](https://mlsanity.com/paper/2409.17455/) (2024)
- [Towards Formalizing Spuriousness of Biased Datasets Using Partial Information Decomposition](https://mlsanity.com/paper/2407.00482/) (2024)
- [Visual Data Diagnosis and Debiasing with Concept Graphs](https://mlsanity.com/paper/2409.18055/) (2024)
- [GeSS: Benchmarking Geometric Deep Learning under Scientific Applications with Distribution Shifts](https://mlsanity.com/paper/2310.08677/) (2023)
- [Reassessing the Validity of Spurious Correlations Benchmarks](https://mlsanity.com/paper/2409.04188/) (2024)
- [Antibody DomainBed: Out-of-Distribution Generalization in Therapeutic Protein Design](https://mlsanity.com/paper/2407.21028/) (2024)
- [Variance Matters: Improving Domain Adaptation via Stratified Sampling](https://mlsanity.com/paper/2512.05226/) (2025)
- [Closed-Form Linear-Probe Dataset Distillation for Pre-trained Vision Models](https://mlsanity.com/paper/2605.07194/) (2026)
- [Robustness May be More Brittle than We Think under Different Degrees of Distribution Shifts](https://mlsanity.com/paper/2310.06622/) (2023)
- [UnLearning from Experience to Avoid Spurious Correlations](https://mlsanity.com/paper/2409.02792/) (2024)
- [MegaCOIN: Enhancing Medium-Grained Color Perception for Vision-Language Models](https://mlsanity.com/paper/2412.03927/) (2024)
- [Superclass-Guided Representation Disentanglement for Spurious Correlation Mitigation](https://mlsanity.com/paper/2508.08570/) (2025)
- [Order Matters: Improving Domain Adaptation by Reordering Data](https://mlsanity.com/paper/2605.05084/) (2026)
- [Teaching Humans Subtle Differences with DIFFusion](https://mlsanity.com/paper/2504.08046/) (2025)
- [On Background Bias of Post-Hoc Concept Embeddings in Computer Vision DNNs](https://mlsanity.com/paper/2504.08602/) (2025)
- [FedDiverse: Tackling Data Heterogeneity in Federated Learning with Diversity-Driven Client Selection](https://mlsanity.com/paper/2504.11216/) (2025)
- [Corner Cases: How Size and Position of Objects Challenge ImageNet-Trained Models](https://mlsanity.com/paper/2505.03569/) (2025)
- [Robust Context-Aware Object Recognition](https://mlsanity.com/paper/2510.00618/) (2025)
- [A2: Smaller Self-Supervised ViTs Localize Better than Larger Ones](https://mlsanity.com/paper/2606.03148/) (2026)

## Abstract (arXiv)

The problem of spurious correlations (SCs) arises when a classifier relies on non-predictive features that happen to be correlated with the labels in the training data. For example, a classifier may misclassify dog breeds based on the background of dog images. This happens when the backgrounds are correlated with other breeds in the training data, leading to misclassifications during test time. Previous SC benchmark datasets suffer from varying issues, e.g., over-saturation or only containing one-to-one (O2O) SCs, but no many-to-many (M2M) SCs arising between groups of spurious attributes and classes. In this paper, we present benchmark-O2O, M2M-Easy, Medium, Hard, an image classification benchmark suite containing spurious correlations between classes and backgrounds. To create this dataset, we employ a text-to-image model to generate photo-realistic images and an image captioning model to filter out unsuitable ones. The resulting dataset is of high quality and contains approximately 152k images. Our experimental results demonstrate that state-of-the-art group robustness methods struggle with benchmark, most notably on the Hard-splits with none of them getting over 70% accuracy on the hardest split using a ResNet50 pretrained on ImageNet. By examining model misclassifications, we detect reliances on spurious backgrounds, demonstrating that our dataset provides a significant challenge.