# A Sober Look at the Robustness of CLIPs to Spurious Features

arXiv: https://arxiv.org/abs/2403.11497 · HTML: https://mlsanity.com/paper/2403.11497/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 41 of its 54 references that are in the map.
- First posted on arXiv 18 Mar 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Prompt learning and adaptation of CLIP](https://mlsanity.com/mllm/prompt-learning-and-adaptation-of-clip/)

## Builds on (most cited first, 30 of 41)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [Evaluating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2305.10355/) (2023)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization](https://mlsanity.com/paper/1911.08731/) (2019)
- [Reproducible scaling laws for contrastive language-image learning](https://mlsanity.com/paper/2212.07143/) (2022)
- [In Search of Lost Domain Generalization](https://mlsanity.com/paper/2007.01434/) (2020)
- [Learning Robust Global Representations by Penalizing Local Predictive Power](https://mlsanity.com/paper/1905.13549/) (2019)
- [Vision-Language Models for Vision Tasks: A Survey](https://mlsanity.com/paper/2304.00685/) (2023)
- [Robust fine-tuning of zero-shot models](https://mlsanity.com/paper/2109.01903/) (2021)
- [Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs](https://mlsanity.com/paper/2401.06209/) (2024)
- [DataComp: In search of the next generation of multimodal datasets](https://mlsanity.com/paper/2304.14108/) (2023)
- [MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning](https://mlsanity.com/paper/2310.09478/) (2023)
- [Measuring Robustness to Natural Distribution Shifts in Image Classification](https://mlsanity.com/paper/2007.00644/) (2020)
- [Noise or Signal: The Role of Image Backgrounds in Object Recognition](https://mlsanity.com/paper/2006.09994/) (2020)
- [An Investigation of Why Overparameterization Exacerbates Spurious Correlations](https://mlsanity.com/paper/2005.04345/) (2020)
- [Data Filtering Networks](https://mlsanity.com/paper/2309.17425/) (2023)
- [Finetune like you pretrain: Improved finetuning of zero-shot vision models](https://mlsanity.com/paper/2212.00638/) (2022)
- [Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)](https://mlsanity.com/paper/2205.01397/) (2022)
- [Identifiability Results for Multimodal Contrastive Learning](https://mlsanity.com/paper/2303.09166/) (2023)
- [Do Image Classifiers Generalize Across Time?](https://mlsanity.com/paper/1906.02168/) (2019)
- [ZIN: When and How to Learn Invariance Without Environment Partition?](https://mlsanity.com/paper/2203.05818/) (2022)
- [Mitigating Spurious Correlations in Multi-modal Models during Fine-tuning](https://mlsanity.com/paper/2304.03916/) (2023)
- [LANCE: Stress-testing Visual Models by Generating Language-guided Counterfactual Images](https://mlsanity.com/paper/2305.19164/) (2023)
- [Model Agnostic Sample Reweighting for Out-of-Distribution Learning](https://mlsanity.com/paper/2301.09819/) (2023)

## Built on by (most cited first, 30 of 38)

- [Exploring Criteria of Loss Reweighting to Enhance LLM Unlearning](https://mlsanity.com/paper/2505.11953/) (2025)
- [Soft Prompt Generation for Domain Generalization](https://mlsanity.com/paper/2404.19286/) (2024)
- [Safety Mirage: How Spurious Correlations Undermine VLM Safety Fine-Tuning and Can Be Mitigated by Machine Unlearning](https://mlsanity.com/paper/2503.11832/) (2025)
- [Are We Done with Object-Centric Learning?](https://mlsanity.com/paper/2504.07092/) (2025)
- [When and How Does CLIP Enable Domain and Compositional Generalization?](https://mlsanity.com/paper/2502.09507/) (2025)
- [From What to How: Attributing CLIP's Latent Components Reveals Unexpected Semantic Reliance](https://mlsanity.com/paper/2505.20229/) (2025)
- [Is Less More? Exploring Token Condensation as Training-free Test-time Adaptation](https://mlsanity.com/paper/2410.14729/) (2024)
- [Debiasing CLIP: Interpreting and Correcting Bias in Attention Heads](https://mlsanity.com/paper/2505.17425/) (2025)
- [Less Precise Can Be More Reliable: A Systematic Evaluation of Quantization's Impact on VLMs Beyond Accuracy](https://mlsanity.com/paper/2509.21173/) (2025)
- [Concept Regions Matter: Benchmarking CLIP with a New Cluster-Importance Approach](https://mlsanity.com/paper/2511.12978/) (2025)
- [ForAug: Mitigating Biases in Image Classification via Controlled Image Compositions](https://mlsanity.com/paper/2503.09399/) (2025)
- [Web Artifact Attacks Disrupt Vision Language Models](https://mlsanity.com/paper/2503.13652/) (2025)
- [CNS-Bench: Benchmarking Image Classifier Robustness Under Continuous Nuisance Shifts](https://mlsanity.com/paper/2507.17651/) (2025)
- [H-SPLID: HSIC-based Saliency Preserving Latent Information Decomposition](https://mlsanity.com/paper/2510.20627/) (2025)
- [Seeing Beyond the Scene: Analyzing and Mitigating Background Bias in Action Recognition](https://mlsanity.com/paper/2512.17953/) (2025)
- [DivIL: Unveiling and Addressing Over-Invariance for Out-of- Distribution Generalization](https://mlsanity.com/paper/2502.12413/) (2025)
- [On the Domain Robustness of Contrastive Vision-Language Models](https://mlsanity.com/paper/2506.23663/) (2025)
- [Robust Context-Aware Object Recognition](https://mlsanity.com/paper/2510.00618/) (2025)
- [WildIng: A Wildlife Image Invariant Representation Model for Geographical Domain Shift](https://mlsanity.com/paper/2601.00993/) (2026)
- [CLIMP: Contrastive Language-Image Mamba Pretraining](https://mlsanity.com/paper/2601.06891/) (2026)
- [Rethinking Brain Decoding with CLIP: The Role of Adversarial Robustness](https://mlsanity.com/paper/2607.03165/) (2026)
- [Doomed to Re-Annotate, Forever: The ImageNet Story](https://mlsanity.com/paper/2608.13783/) (2026)
- [ANCHOR: LLM-driven Subject Conditioning for Text-to-Image Synthesis](https://mlsanity.com/paper/2404.10141/) (2024)
- [Refining Skewed Perceptions in Vision-Language Contrastive Models through Visual Representations](https://mlsanity.com/paper/2405.14030/) (2024)
- [Bringing the Context Back into Object Recognition, Robustly](https://mlsanity.com/paper/2411.15933/) (2024)
- [Unleashing the Power of Pre-trained Encoders for Universal Adversarial Attack Detection](https://mlsanity.com/paper/2504.00429/) (2025)
- [Impact of Pretraining Word Co-occurrence on Compositional Generalization in Multimodal Models](https://mlsanity.com/paper/2507.08000/) (2025)
- [A Matter of Time: Revealing the Structure of Time in Vision-Language Models](https://mlsanity.com/paper/2510.19559/) (2025)
- [Turning Adversaries into Allies: Reversing Typographic Attacks for Multimodal E-Commerce Product Retrieval](https://mlsanity.com/paper/2511.05325/) (2025)
- [MeGU: Machine-Guided Unlearning with Target Feature Disentanglement](https://mlsanity.com/paper/2602.17088/) (2026)

## Abstract (arXiv)

Large vision language models, such as CLIP, demonstrate impressive robustness to spurious features than single-modal models trained on ImageNet. However, existing test datasets are typically curated based on ImageNet-trained models, which aim to capture the spurious features inherited in ImageNet. Benchmarking CLIP models based on the ImageNet-oriented spurious features may not be sufficient to reflect the extent to which CLIP models are robust to spurious correlations within CLIP training data, e.g., LAION. To this end, we craft a new challenging dataset named CounterAnimal designed to reveal the reliance of CLIP models on realistic spurious features. Specifically, we split animal photos into groups according to the backgrounds, and then identify a pair of groups for each class where a CLIP model shows high-performance drops across the two groups. Our evaluations show that the spurious features captured by CounterAnimal are generically learned by CLIP models with different backbones and pre-train data, yet have limited influence for ImageNet models. We provide theoretical insights that the CLIP objective cannot offer additional robustness. Furthermore, we also re-evaluate strategies such as scaling up parameters and high-quality pre-trained data. We find that they still help mitigate the spurious features, providing a promising path for future developments.