mlsanityData to 7 Oct 2026

Spawrious: A Benchmark for Fine Control of Spurious Correlation Biases

arXiv 2303.05470 ↗First posted 9 Mar 2023cs.CV cs.LGcited by 41

Not on the map. Why?

Abstract (arXiv)

The problem of spurious correlations (SCs) arises when a classifier relies on non-predictive features that happen to be correlated with the labels in the training data. For example, a classifier may misclassify dog breeds based on the background of dog images. This happens when the backgrounds are correlated with other breeds in the training data, leading to misclassifications during test time. Previous SC benchmark datasets suffer from varying issues, e.g., over-saturation or only containing one-to-one (O2O) SCs, but no many-to-many (M2M) SCs arising between groups of spurious attributes and classes. In this paper, we present benchmark-O2O, M2M-Easy, Medium, Hard, an image classification benchmark suite containing spurious correlations between classes and backgrounds. To create this dataset, we employ a text-to-image model to generate photo-realistic images and an image captioning model to filter out unsuitable ones. The resulting dataset is of high quality and contains approximately 152k images. Our experimental results demonstrate that state-of-the-art group robustness methods struggle with benchmark, most notably on the Hard-splits with none of them getting over 70% accuracy on the hardest split using a ResNet50 pretrained on ImageNet. By examining model misclassifications, we detect reliances on spurious backgrounds, demonstrating that our dataset provides a significant challenge.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Builds on 81 of its 123 references that are in the map. First posted on arXiv 9 Mar 2023.

Builds on

81 of 123 references.

References · 81

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  6. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  7. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  8. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  9. 2015Learning Deep Features for Discriminative Localizationcited by 2,430
  10. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  11. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  12. 2014Unsupervised Domain Adaptation by Backpropagationcited by 1,906
38 more
  1. 2019Invariant Risk Minimizationcited by 1,832
  2. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
  3. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  4. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  5. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,232
  6. 2016Deep CORAL: Correlation Alignment for Deep Domain Adaptationcited by 1,194
  7. 2020In Search of Lost Domain Generalizationcited by 1,042
  8. 2020Out-of-Distribution Generalization via Risk Extrapolation (REx)cited by 737
  9. 2023Extracting Training Data from Diffusion Modelscited by 723
  10. 2013Domain Generalization via Invariant Feature Representationcited by 622
  11. 2015Causal inference using invariant prediction: identification and confidence intervalscited by 533
  12. 2021Just Train Twice: Improving Group Robustness without Training Group Informationcited by 522
  13. 2020Underspecification Presents Challenges for Credibility in Modern Machine Learningcited by 488
  14. 2017Meta-learners for Estimating Heterogeneous Treatment Effects using Machine Learningcited by 475
  15. 2021SWAD: Domain Generalization by Seeking Flat Minimacited by 431
  16. 2019Domain Generalization via Model-Agnostic Learning of Semantic Featurescited by 405
  17. 2022Last Layer Re-Training is Sufficient for Robustness to Spurious Correlationscited by 379
  18. 2020Noise or Signal: The Role of Image Backgrounds in Object Recognitioncited by 342
  19. 2016Unsupervised Domain Adaptation with Residual Transfer Networkscited by 340
  20. 2017Functional Map of the Worldcited by 338
  21. 2017Quasi-Oracle Estimation of Heterogeneous Treatment Effectscited by 328
  22. 2020An Investigation of Why Overparameterization Exacerbates Spurious Correlationscited by 310
  23. 2020Environment Inference for Invariant Learningcited by 306
  24. 2020The Risks of Invariant Risk Minimizationcited by 262
  25. 2021Gradient Matching for Domain Generalizationcited by 258
  26. 2019Adversarial Domain Adaptation with Domain Mixupcited by 250
  27. 2019Episodic Training for Domain Generalizationcited by 245
  28. 2018Generalisation in humans and deep neural networkscited by 237
  29. 2021Improving Robustness using Generated Datacited by 233
  30. 2020Domain Generalization using Causal Matchingcited by 224
  31. 2020No Subclass Left Behind: Fine-Grained Robustness in Coarse-Grained Classification Problemscited by 208
  32. 2022Surgical Fine-Tuning Improves Adaptation to Distribution Shiftscited by 205
  33. 2022Improving Out-of-Distribution Robustness via Selective Augmentationcited by 202
  34. 2021Fishr: Invariant Gradient Variances for Out-of-Distribution Generalizationcited by 186
  35. 2021Simple data balancing achieves competitive worst-group-accuracycited by 177
  36. 2021A Fine-Grained Analysis on Distribution Shiftcited by 171
  37. 2022Diverse Weight Averaging for Out-of-Distribution Generalizationcited by 148
  38. 2023Debiasing Vision-Language Models via Biased Promptscited by 147

50 most cited of 81 All 81 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 11; 2025: 13; 2026: 9.

41 papers.

Sort:

By area:

  1. 2023Challenges and Applications of Large Language Modelscited by 251
  2. 2025REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewardscited by 121
  3. 2023The MiniPile Challenge for Data-Efficient Language Modelscited by 74
  4. 2024A Multimodal Automated Interpretability Agentcited by 62
  5. 2023COCO-Counterfactuals: Automatically Constructed Counterfactual Examples for Image-Text Pairscited by 39
  6. 2023Machine Vision Therapy: Multimodal Large Language Models Can Enhance Visual Robustness via Denoising In-Context Learningcited by 27
  7. 2024Navigating Shortcuts, Spurious Correlations, and Confounders: From Origins via Detection to Mitigationcited by 22
  8. 2023Rethinking the Evaluation Protocol of Domain Generalizationcited by 20
  9. 2024A Survey on Evaluation of Out-of-Distribution Generalizationcited by 20
  10. 2023TTIDA: Controllable Generative Data Augmentation via Text-to-Text and Text-to-Image Modelscited by 14
  11. 2025Escaping the SpuriVerse: Can Large Vision-Language Models Generalize Beyond Seen Spurious Correlations?cited by 13
  12. 2024Navigating the Shortcut Maze: A Comprehensive Analysis of Shortcut Learning in Text Classification by Language Modelscited by 10
29 more
  1. 2024Towards Formalizing Spuriousness of Biased Datasets Using Partial Information Decompositioncited by 6
  2. 2024Visual Data Diagnosis and Debiasing with Concept Graphscited by 6
  3. 2023GeSS: Benchmarking Geometric Deep Learning under Scientific Applications with Distribution Shiftscited by 4
  4. 2024Reassessing the Validity of Spurious Correlations Benchmarkscited by 4
  5. 2024Antibody DomainBed: Out-of-Distribution Generalization in Therapeutic Protein Designcited by 3
  6. 2025Variance Matters: Improving Domain Adaptation via Stratified Samplingcited by 3
  7. 2023Robustness May be More Brittle than We Think under Different Degrees of Distribution Shiftscited by 2
  8. 2024UnLearning from Experience to Avoid Spurious Correlationscited by 2
  9. 2024MegaCOIN: Enhancing Medium-Grained Color Perception for Vision-Language Modelscited by 2
  10. 2025Superclass-Guided Representation Disentanglement for Spurious Correlation Mitigationcited by 2
  11. 2026Order Matters: Improving Domain Adaptation by Reordering Datacited by 2
  12. 2025Teaching Humans Subtle Differences with DIFFusioncited by 1
  13. 2025On Background Bias of Post-Hoc Concept Embeddings in Computer Vision DNNscited by 1
  14. 2025FedDiverse: Tackling Data Heterogeneity in Federated Learning with Diversity-Driven Client Selectioncited by 1
  15. 2025Corner Cases: How Size and Position of Objects Challenge ImageNet-Trained Modelscited by 1
  16. 2025Robust Context-Aware Object Recognitioncited by 1
  17. 2026Closed-Form Linear-Probe Dataset Distillation for Pre-trained Vision Modelscited by 1
  18. 2026A2: Smaller Self-Supervised ViTs Localize Better than Larger Onescited by 1
  19. 2026Automated Background Swapping for Robustness against Spurious Backgroundscited by 1
  20. 2024Bringing the Context Back into Object Recognition, Robustlycited by 0
  21. 2025Do Vision Models Develop Human-Like Progressive Difficulty Understanding?cited by 0
  22. 2025Like Oil and Water: Group Robustness Methods and Poisoning Defenses May Be at Oddscited by 0
  23. 2025SpurBreast: A Curated Dataset for Investigating Spurious Correlations in Real-world Breast MRI Classificationcited by 0
  24. 2025Spatially Grounded Concept-Based Image Classificationcited by 0
  25. 2026eXplaining to Learn (eX2L): Regularization Using Contrastive Visual Explanation Pairs for Distribution Shiftscited by 0
  26. 2026Deep Attention Reweighting: Post-Hoc Attention-Based Feature Aggregation in CNNs for Disentangling Core and Spurious Features under Spurious Correlationscited by 0
  27. 2026ContextShift: A Controlled Benchmark for Context Dependence in Object Detectioncited by 0
  28. 2026Variance-reduced Domain Adaptation using Paired Samplingcited by 0
  29. 2026Online Variance Reduction for Domain Adaptation on Streaming Datacited by 0