mlsanityData to 7 Oct 2026

Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)

arXiv 2205.01397 ↗First posted 3 May 2022cs.CV cs.CL cs.LGcited by 137

Not on the map. Why?

Cited by 30 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Contrastively trained language-image models such as CLIP, ALIGN, and BASIC have demonstrated unprecedented robustness to multiple challenging natural distribution shifts. Since these language-image models differ from previous training approaches in several ways, an important question is what causes the large robustness gains. We answer this question via a systematic experimental investigation. Concretely, we study five different possible causes for the robustness gains: (i) the training set size, (ii) the training distribution, (iii) language supervision at training time, (iv) language supervision at test time, and (v) the contrastive loss function. Our experiments show that the more diverse training distribution is the main cause for the robustness gains, with the other factors contributing little to no robustness. Beyond our experimental results, we also introduce ImageNet-Captions, a version of ImageNet with original text annotations from Flickr, to enable further controlled experiments of language-image training.

Key facts. Cited by 137 papers in the map (data through 7 Oct 2026). Builds on 32 of its 41 references that are in the map. First posted on arXiv 3 May 2022.

Builds on

32 of 41 references.

References · 32

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  8. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  9. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  10. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  11. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  12. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
20 more
  1. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  2. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  3. 2019Natural Adversarial Examplescited by 1,401
  4. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  5. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  6. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  7. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  8. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  9. 2020Measuring Robustness to Natural Distribution Shifts in Image Classificationcited by 452
  10. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
  11. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  12. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 285
  13. 2021Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalizationcited by 248
  14. 2021Combined Scaling for Zero-shot Transfer Learningcited by 182
  15. 2021RedCaps: web-curated image-text data created by the people, for the peoplecited by 175
  16. 2020Learning Visual Representations with Caption Annotationscited by 123
  17. 2020On Robustness and Transferability of Convolutional Neural Networkscited by 98
  18. 2021The Evolution of Out-of-Distribution Robustness Throughout Fine-Tuningcited by 74
  19. 2020Early Stopping in Deep Networks: Double Descent and How to Eliminate itcited by 40
  20. 2021Does language help generalization in vision models?cited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 13; 2023: 46; 2024: 42; 2025: 26; 2026: 10.

137 papers.

Sort:

By area:

  1. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  2. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  3. 2023TRAK: Attributing Model Behavior at Scalecited by 273
  4. 2023Data Filtering Networkscited by 247
  5. 2023Revisiting Class-Incremental Learning with Pre-Trained Models: Generalizability and Adaptivity are All You Needcited by 183
  6. 2023BioCLIP: A Vision Foundation Model for the Tree of Lifecited by 159
  7. 2025Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectivescited by 120
  8. 2022SuS-X: Training-Free Name-Only Transfer of Vision-Language Modelscited by 118
  9. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  10. 2022Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIPcited by 97
  11. 2022Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalizationcited by 97
  12. 2023MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Trainingcited by 96
38 more
  1. 2023CLIP-DINOiser: Teaching CLIP a few DINO tricks for open-vocabulary semantic segmentationcited by 94
  2. 2023Improving Multimodal Datasets with Image Captioningcited by 93
  3. 2022A Whac-A-Mole Dilemma: Shortcuts Come in Multiples Where Mitigating One Amplifies Otherscited by 85
  4. 2023A Comprehensive Study on Robustness of Image Classification Models: Benchmarking and Rethinkingcited by 85
  5. 2024Manipulate-Anything: Automating Real-World Robots using Vision-Language Modelscited by 82
  6. 2024No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performancecited by 75
  7. 2024Scaling Laws for Data Filtering -- Data Curation cannot be Compute Agnosticcited by 74
  8. 2024AbdomenAtlas: A Large-Scale, Detailed-Annotated, & Multi-Center Dataset for Efficient Transfer Learning and Open Algorithmic Benchmarkingcited by 64
  9. 2025Scaling Language-Free Visual Representation Learningcited by 62
  10. 2022PODA: Prompt-driven Zero-shot Domain Adaptationcited by 60
  11. 2023Discovering and Mitigating Visual Biases through Keyword Explanationcited by 58
  12. 2022ID and OOD Performance Are Sometimes Inversely Correlated on Real-world Datasetscited by 44
  13. 2024A Closer Look at the Robustness of Contrastive Language-Image Pre-Training (CLIP)cited by 44
  14. 2023T-MARS: Improving Visual Representations by Circumventing Text Feature Learningcited by 43
  15. 2022Is a Caption Worth a Thousand Images? A Controlled Study for Representation Learningcited by 39
  16. 2024Benchmarking and Improving Bird's Eye View Perception Robustness in Autonomous Drivingcited by 39
  17. 2024A Sober Look at the Robustness of CLIPs to Spurious Featurescited by 38
  18. 2023Does CLIP's Generalization Performance Mainly Stem from High Train-Test Similarity?cited by 35
  19. 2022FedTune: A Deep Dive into Efficient Federated Fine-Tuning with Pre-trained Transformerscited by 33
  20. 2022Context-Aware Robust Fine-Tuningcited by 27
  21. 2023A Simple Recipe for Language-guided Domain Generalized Segmentationcited by 27
  22. 2023On the Connection between Pre-training Data Diversity and Fine-tuning Robustnesscited by 25
  23. 2024Application-Driven Innovation in Machine Learningcited by 24
  24. 2023Robust Visual Question Answering: Datasets, Methods, and Future Challengescited by 23
  25. 2023MLLMs-Augmented Visual-Language Representation Learningcited by 21
  26. 2025X-Transfer Attacks: Towards Super Transferable Adversarial Attacks on CLIPcited by 21
  27. 2024An Empirical Study Into What Matters for Calibrating Vision-Language Modelscited by 20
  28. 2024The Unmet Promise of Synthetic Training Images: Using Retrieved Real Images Performs Bettercited by 19
  29. 2024Active Data Curation Effectively Distills Large-Scale Multimodal Modelscited by 19
  30. 2023The Role of Pre-training Data in Transfer Learningcited by 17
  31. 2023Domain Aligned CLIP for Few-shot Classificationcited by 16
  32. 2024On the Surprising Effectiveness of Attention Transfer for Vision Transformerscited by 16
  33. 2024On Catastrophic Inheritance of Large Foundation Modelscited by 15
  34. 2024A Survey on Safe Multi-Modal Learning Systemcited by 15
  35. 2025ProKeR: A Kernel Perspective on Few-Shot Adaptation of Large Vision-Language Modelscited by 15
  36. 2023Benchmarking Zero-Shot Recognition with Vision-Language Models: Challenges on Granularity and Specificitycited by 14
  37. 2023Effective Robustness against Natural Distribution Shifts for Models with Different Training Datacited by 13
  38. 2023Unified Out-Of-Distribution Detection: A Model-Specific Perspectivecited by 13

50 most cited of 137 All 137 (JSON)