mlsanityData to 7 Oct 2026

A Sober Look at the Robustness of CLIPs to Spurious Features

arXiv 2403.11497 ↗First posted 18 Mar 2024cs.CV cs.LG stat.MLcited by 38

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Large vision language models, such as CLIP, demonstrate impressive robustness to spurious features than single-modal models trained on ImageNet. However, existing test datasets are typically curated based on ImageNet-trained models, which aim to capture the spurious features inherited in ImageNet. Benchmarking CLIP models based on the ImageNet-oriented spurious features may not be sufficient to reflect the extent to which CLIP models are robust to spurious correlations within CLIP training data, e.g., LAION. To this end, we craft a new challenging dataset named CounterAnimal designed to reveal the reliance of CLIP models on realistic spurious features. Specifically, we split animal photos into groups according to the backgrounds, and then identify a pair of groups for each class where a CLIP model shows high-performance drops across the two groups. Our evaluations show that the spurious features captured by CounterAnimal are generically learned by CLIP models with different backbones and pre-train data, yet have limited influence for ImageNet models. We provide theoretical insights that the CLIP objective cannot offer additional robustness. Furthermore, we also re-evaluate strategies such as scaling up parameters and high-quality pre-trained data. We find that they still help mitigate the spurious features, providing a promising path for future developments.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 41 of its 54 references that are in the map. First posted on arXiv 18 Mar 2024.

Builds on

41 of 54 references.

References · 41

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  6. 2019Invariant Risk Minimizationcited by 1,832
  7. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  8. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  9. 2019Natural Adversarial Examplescited by 1,401
  10. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,232
  11. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  12. 2020In Search of Lost Domain Generalizationcited by 1,042
29 more
  1. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  2. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  3. 2021Robust fine-tuning of zero-shot modelscited by 785
  4. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  5. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  6. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  7. 2020Measuring Robustness to Natural Distribution Shifts in Image Classificationcited by 452
  8. 2020Noise or Signal: The Role of Image Backgrounds in Object Recognitioncited by 342
  9. 2020An Investigation of Why Overparameterization Exacerbates Spurious Correlationscited by 310
  10. 2023Data Filtering Networkscited by 247
  11. 2022Finetune like you pretrain: Improved finetuning of zero-shot vision modelscited by 149
  12. 2022Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)cited by 137
  13. 2019Do Image Classifiers Generalize Across Time?cited by 59
  14. 2022ZIN: When and How to Learn Invariance Without Environment Partition?cited by 58
  15. 2023Identifiability Results for Multimodal Contrastive Learningcited by 58
  16. 2023Mitigating Spurious Correlations in Multi-modal Models during Fine-tuningcited by 47
  17. 2023LANCE: Stress-testing Visual Models by Generating Language-guided Counterfactual Imagescited by 46
  18. 2023Model Agnostic Sample Reweighting for Out-of-Distribution Learningcited by 43
  19. 2023Understanding Multimodal Contrastive Learning and Incorporating Unpaired Datacited by 43
  20. 2021Finding and Fixing Spurious Patterns with Explanationscited by 38
  21. 2022Pareto Invariant Risk Minimization: Towards Mitigating the Optimization Dilemma in Out-of-Distribution Generalizationcited by 38
  22. 2023Dataset Interfaces: Diagnosing Model Failures Using Controllable Counterfactual Generationcited by 35
  23. 2023Understanding and Improving Feature Learning for Out-of-Distribution Generalizationcited by 35
  24. 2023Does CLIP's Generalization Performance Mainly Stem from High Train-Test Similarity?cited by 35
  25. 2023ImageNet-E: Benchmarking Neural Network Robustness via Attribute Editingcited by 30
  26. 2023On the Generalization of Multi-modal Contrastive Learningcited by 30
  27. 2023Spurious Feature Diversification Improves Out-of-distribution Generalizationcited by 26
  28. 2023Effective Robustness against Natural Distribution Shifts for Models with Different Training Datacited by 13
  29. 2023Continuous Invariance Learningcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 20; 2026: 13.

38 papers.

Sort:

By area:

  1. 2025Exploring Criteria of Loss Reweighting to Enhance LLM Unlearningcited by 39
  2. 2024Soft Prompt Generation for Domain Generalizationcited by 24
  3. 2025Safety Mirage: How Spurious Correlations Undermine VLM Safety Fine-Tuning and Can Be Mitigated by Machine Unlearningcited by 16
  4. 2025Are We Done with Object-Centric Learning?cited by 10
  5. 2025When and How Does CLIP Enable Domain and Compositional Generalization?cited by 9
  6. 2024Is Less More? Exploring Token Condensation as Training-free Test-time Adaptationcited by 5
  7. 2025Debiasing CLIP: Interpreting and Correcting Bias in Attention Headscited by 4
  8. 2025From What to How: Attributing CLIP's Latent Components Reveals Unexpected Semantic Reliancecited by 4
  9. 2025Less Precise Can Be More Reliable: A Systematic Evaluation of Quantization's Impact on VLMs Beyond Accuracycited by 3
  10. 2025ForAug: Mitigating Biases in Image Classification via Controlled Image Compositionscited by 2
  11. 2025Web Artifact Attacks Disrupt Vision Language Modelscited by 2
  12. 2025CNS-Bench: Benchmarking Image Classifier Robustness Under Continuous Nuisance Shiftscited by 2
26 more
  1. 2025H-SPLID: HSIC-based Saliency Preserving Latent Information Decompositioncited by 2
  2. 2025Concept Regions Matter: Benchmarking CLIP with a New Cluster-Importance Approachcited by 2
  3. 2025Seeing Beyond the Scene: Analyzing and Mitigating Background Bias in Action Recognitioncited by 2
  4. 2025DivIL: Unveiling and Addressing Over-Invariance for Out-of- Distribution Generalizationcited by 1
  5. 2025On the Domain Robustness of Contrastive Vision-Language Modelscited by 1
  6. 2025Robust Context-Aware Object Recognitioncited by 1
  7. 2026CLIMP: Contrastive Language-Image Mamba Pretrainingcited by 1
  8. 2026Rethinking Brain Decoding with CLIP: The Role of Adversarial Robustnesscited by 1
  9. 2026Doomed to Re-Annotate, Forever: The ImageNet Storycited by 1
  10. 2024ANCHOR: LLM-driven Subject Conditioning for Text-to-Image Synthesiscited by 0
  11. 2024Refining Skewed Perceptions in Vision-Language Contrastive Models through Visual Representationscited by 0
  12. 2024Bringing the Context Back into Object Recognition, Robustlycited by 0
  13. 2025Unleashing the Power of Pre-trained Encoders for Universal Adversarial Attack Detectioncited by 0
  14. 2025Impact of Pretraining Word Co-occurrence on Compositional Generalization in Multimodal Modelscited by 0
  15. 2025A Matter of Time: Revealing the Structure of Time in Vision-Language Modelscited by 0
  16. 2025Turning Adversaries into Allies: Reversing Typographic Attacks for Multimodal E-Commerce Product Retrievalcited by 0
  17. 2026WildIng: A Wildlife Image Invariant Representation Model for Geographical Domain Shiftcited by 0
  18. 2026MeGU: Machine-Guided Unlearning with Target Feature Disentanglementcited by 0
  19. 2026The Invisible Gorilla Effect in Out-of-distribution Detectioncited by 0
  20. 2026Is CLIP Cross-Eyed? Revealing and Mitigating Center Bias in the CLIP Familycited by 0
  21. 2026Metonymy in vision models undermines attention-based interpretabilitycited by 0
  22. 2026Birds of a Feather Flock Together: Background-Invariant Representations via Linear Structure in VLMscited by 0
  23. 2026Neutral-Reference Prompting for Vision-Language Modelscited by 0
  24. 2026The Hyperspherical Geometry of CLIP Latent Space: A Semantic Mixture Modelcited by 0
  25. 2026Invisible Shortcuts: Why Vision Encoders Know Your Cameracited by 0
  26. 2026Modelpedia: A Catalog of Model Findings for the Meta-Science of AIcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Calibrating Multi-modal Representations: A Pursuit of Group Robustness without Annotations
  2. 2024Toward a Holistic Evaluation of Robustness in CLIP Models
  3. 2024The Hard Positive Truth about Vision-Language Compositionality
  4. 2025SAGE: Spuriousness-Aware Guided Prompt Exploration for Mitigating Multimodal Bias
  5. 2026Scaling Vision-Language Models Is Not Enough to Mitigate Bias
  6. 2023Understanding the Vulnerability of CLIP to Image Compression
  7. 2025SegDebias: Test-Time Bias Mitigation for ViT-Based CLIP via Segmentation
  8. 2026Beyond Standard Benchmarks: A Systematic Audit of Vision-Language Model's Robustness to Natural Semantic Variation Across Diverse Tasks