# Does CLIP's Generalization Performance Mainly Stem from High Train-Test Similarity?

arXiv: https://arxiv.org/abs/2310.09562 · HTML: https://mlsanity.com/paper/2310.09562/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 32 references that are in the map.
- First posted on arXiv 14 Oct 2023.

## Builds on (most cited first, 26 of 26)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization](https://mlsanity.com/paper/1911.08731/) (2019)
- [Learning Robust Global Representations by Penalizing Local Predictive Power](https://mlsanity.com/paper/1905.13549/) (2019)
- [DataComp: In search of the next generation of multimodal datasets](https://mlsanity.com/paper/2304.14108/) (2023)
- [Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling](https://mlsanity.com/paper/2111.03930/) (2021)
- [Functional Map of the World](https://mlsanity.com/paper/1711.07846/) (2017)
- [DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data](https://mlsanity.com/paper/2306.09344/) (2023)
- [SemDeDup: Data-efficient learning at web-scale through semantic deduplication](https://mlsanity.com/paper/2303.09540/) (2023)
- [Combined Scaling for Zero-shot Transfer Learning](https://mlsanity.com/paper/2111.10050/) (2021)
- [Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)](https://mlsanity.com/paper/2205.01397/) (2022)
- [On Interaction Between Augmentations and Corruptions in Natural Corruption Robustness](https://mlsanity.com/paper/2102.11273/) (2021)
- [Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIP](https://mlsanity.com/paper/2208.05516/) (2022)
- [Divide and Contrast: Self-supervised Learning from Uncurated Data](https://mlsanity.com/paper/2105.08054/) (2021)
- [Compositional Generalization from First Principles](https://mlsanity.com/paper/2307.05596/) (2023)
- [Is a Caption Worth a Thousand Images? A Controlled Study for Representation Learning](https://mlsanity.com/paper/2207.07635/) (2022)

## Built on by (most cited first, 30 of 35)

- [CLIP-DINOiser: Teaching CLIP a few DINO tricks for open-vocabulary semantic segmentation](https://mlsanity.com/paper/2312.12359/) (2023)
- [No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance](https://mlsanity.com/paper/2404.04125/) (2024)
- [Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Models](https://mlsanity.com/paper/2404.07983/) (2024)
- [A Sober Look at the Robustness of CLIPs to Spurious Features](https://mlsanity.com/paper/2403.11497/) (2024)
- [Frustratingly Easy Test-Time Adaptation of Vision-Language Models](https://mlsanity.com/paper/2405.18330/) (2024)
- [Active Data Curation Effectively Distills Large-Scale Multimodal Models](https://mlsanity.com/paper/2411.18674/) (2024)
- [What Makes CLIP More Robust to Long-Tailed Pre-Training Data? A Controlled Study for Transferable Insights](https://mlsanity.com/paper/2405.21070/) (2024)
- [Feature contamination: Neural networks learn uncorrelated features and fail to generalize](https://mlsanity.com/paper/2406.03345/) (2024)
- [Robustness of Deep Learning for Accelerated MRI: Benefits of Diverse Training Data](https://mlsanity.com/paper/2312.10271/) (2023)
- [On the Comparison between Multi-modal and Single-modal Contrastive Learning](https://mlsanity.com/paper/2411.02837/) (2024)
- [MATH-Beyond: A Benchmark for RL to Expand Beyond the Base Model](https://mlsanity.com/paper/2510.11653/) (2025)
- [Understanding and Mitigating Miscalibration in Prompt Tuning for Vision-Language Models](https://mlsanity.com/paper/2410.02681/) (2024)
- [Is Large-Scale Pretraining the Secret to Good Domain Generalization?](https://mlsanity.com/paper/2412.02856/) (2024)
- [When and How Does CLIP Enable Domain and Compositional Generalization?](https://mlsanity.com/paper/2502.09507/) (2025)
- [Position: An Empirically Grounded Identifiability Theory Will Accelerate Self-Supervised Learning Research](https://mlsanity.com/paper/2504.13101/) (2025)
- [Is Less More? Exploring Token Condensation as Training-free Test-time Adaptation](https://mlsanity.com/paper/2410.14729/) (2024)
- [Semantic Document Derendering: SVG Reconstruction via Vision-Language Modeling](https://mlsanity.com/paper/2511.13478/) (2025)
- [Explaining Caption-Image Interactions in CLIP Models with Second-Order Attributions](https://mlsanity.com/paper/2408.14153/) (2024)
- [Parameter-efficient Bayesian Neural Networks for Uncertainty-aware Depth Estimation](https://mlsanity.com/paper/2409.17085/) (2024)
- [LAION-C: An Out-of-Distribution Benchmark for Web-Scale Vision Models](https://mlsanity.com/paper/2506.16950/) (2025)
- [Less Precise Can Be More Reliable: A Systematic Evaluation of Quantization's Impact on VLMs Beyond Accuracy](https://mlsanity.com/paper/2509.21173/) (2025)
- [Synergy and Diversity in CLIP: Enhancing Performance Through Adaptive Backbone Ensembling](https://mlsanity.com/paper/2405.17139/) (2024)
- [Automatic benchmarking of large multimodal models via iterative experiment programming](https://mlsanity.com/paper/2406.12321/) (2024)
- [FOCoOp: Enhancing Out-of-Distribution Robustness in Federated Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2506.16218/) (2025)
- [Bridging Domains through Subspace-Aware Model Merging](https://mlsanity.com/paper/2603.05768/) (2026)
- [Imperfect Vision Encoders: Efficient and Robust Tuning for Vision-Language Models](https://mlsanity.com/paper/2407.16526/) (2024)
- [MMAPG: A Training-Free Framework for Multimodal Multi-hop Question Answering via Adaptive Planning Graphs](https://mlsanity.com/paper/2508.16051/) (2025)
- [Fair Context Learning for Evidence-Balanced Test-Time Adaptation in Vision-Language Models](https://mlsanity.com/paper/2602.07027/) (2026)
- [Do Vision Models Develop Human-Like Progressive Difficulty Understanding?](https://mlsanity.com/paper/2503.13058/) (2025)
- [FindMeIfYouCan: Bringing Open Set metrics to textitnear ,  textitfar  and textitfarther Out-of-Distribution Object Detection](https://mlsanity.com/paper/2506.14008/) (2025)

## Abstract (arXiv)

Foundation models like CLIP are trained on hundreds of millions of samples and effortlessly generalize to new tasks and inputs. Out of the box, CLIP shows stellar zero-shot and few-shot capabilities on a wide range of out-of-distribution (OOD) benchmarks, which prior works attribute mainly to today's large and comprehensive training dataset (like LAION). However, it is questionable how meaningful terms like out-of-distribution generalization are for CLIP as it seems likely that web-scale datasets like LAION simply contain many samples that are similar to common OOD benchmarks originally designed for ImageNet. To test this hypothesis, we retrain CLIP on pruned LAION splits that replicate ImageNet's train-test similarity with respect to common OOD benchmarks. While we observe a performance drop on some benchmarks, surprisingly, CLIP's overall performance remains high. This shows that high train-test similarity is insufficient to explain CLIP's OOD performance, and other properties of the training data must drive CLIP to learn more generalizable representations. Additionally, by pruning data points that are dissimilar to the OOD benchmarks, we uncover a 100M split of LAION (frac14th of its original size) on which CLIP can be trained to match its original OOD performance.