mlsanityData to 14 Oct 2026

SLIP: Self-supervision meets Language-Image Pre-training

arXiv 2112.12750 ↗First posted 23 Dec 2021cs.CVcited by 436

Not on the map. Why?

Cited by 148 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent work has shown that self-supervised pre-training leads to improvements over supervised learning on challenging visual recognition tasks. CLIP, an exciting new approach to learning with language supervision, demonstrates promising performance on a wide variety of benchmarks. In this work, we explore whether self-supervised learning can aid in the use of language supervision for visual representation learning. We introduce SLIP, a multi-task learning framework for combining self-supervised learning and CLIP pre-training. After pre-training with Vision Transformers, we thoroughly evaluate representation quality and compare performance to both CLIP and self-supervised learning under three distinct settings: zero-shot transfer, linear classification, and end-to-end finetuning. Across ImageNet and a battery of additional datasets, we find that SLIP improves accuracy by a large margin. We validate our results further with experiments on different model sizes, training schedules, and pre-training datasets. Our findings show that SLIP enjoys the best of both worlds: better performance than self-supervision (+8.1% linear accuracy) and language supervision (+5.2% zero-shot accuracy).

Key facts. Cited by 436 papers in the map (data through 14 Oct 2026). Builds on 39 of its 67 references that are in the map. First posted on arXiv 23 Dec 2021.

Builds on

39 of 67 references.

References · 39

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  9. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  10. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  11. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  12. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
27 more
  1. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  2. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,162
  3. 2013Describing Textures in the Wildcited by 2,125
  4. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  5. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  6. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  7. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  8. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  9. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  10. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,028
  11. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 797
  12. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 490
  13. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 425
  14. 2018Rotation Equivariant CNNs for Digital Pathologycited by 422
  15. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  16. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 339
  17. 2021Multimodal datasets: misogyny, pornography, and malignant stereotypescited by 327
  18. 2019A Short Note on the Kinetics-700 Human Action Datasetcited by 310
  19. 2013Challenges in Representation Learning: A report on three machine learning contestscited by 302
  20. 2019Scaling and Benchmarking Self-Supervised Visual Representation Learningcited by 201
  21. 2015Learning Visual Features from Large Weakly Supervised Datacited by 165
  22. 2021Multimodal Contrastive Training for Visual Representation Learningcited by 124
  23. 2020Learning Visual Representations with Caption Annotationscited by 123
  24. 2021Are Large-scale Datasets Necessary for Self-Supervised Pre-training?cited by 117
  25. 2016Learning Visual N-Grams from Web Datacited by 95
  26. 2021Divide and Contrast: Self-supervised Learning from Uncurated Datacited by 85
  27. 2021MURAL: Multimodal, Multitask Retrieval Across Languagescited by 44

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 76; 2023: 126; 2024: 124; 2025: 72; 2026: 38.

436 papers.

Sort:

By area:

  1. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  2. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  3. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  4. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  5. 2022Multimodal Learning with Transformers: A Surveycited by 400
  6. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Modelscited by 375
  7. 2023RemoteCLIP: A Vision Language Foundation Model for Remote Sensingcited by 363
  8. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  9. 2022ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understandingcited by 267
  10. 2023A Cookbook of Self-Supervised Learningcited by 250
  11. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  12. 2023Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text Aligned Latent Representationcited by 228
38 more
  1. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 218
  2. 2023Improving CLIP Training with Language Rewritescited by 217
  3. 2023Scalable 3D Captioning with Pretrained Modelscited by 203
  4. 2023ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understandingcited by 202
  5. 2022MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretrainingcited by 187
  6. 2023StableRep: Synthetic Images from Text-to-Image Models Make Strong Visual Representation Learnerscited by 177
  7. 2024Probing the 3D Awareness of Visual Foundation Modelscited by 177
  8. 2023Towards Open Vocabulary Learning: A Surveycited by 166
  9. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  10. 2023Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learnerscited by 152
  11. 2023RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote Sensingcited by 151
  12. 2022ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Modelscited by 141
  13. 2022Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)cited by 138
  14. 2022KNN-Diffusion: Image Generation via Large-Scale Retrievalcited by 137
  15. 2023Hyperbolic Image-Text Representationscited by 136
  16. 2023Weakly Supervised 3D Open-vocabulary Segmentationcited by 135
  17. 2023Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Imagescited by 134
  18. 2024Exploring scalable medical image encoders beyond text supervisioncited by 132
  19. 2022CyCLIP: Cyclic Contrastive Language-Image Pretrainingcited by 127
  20. 2022Task Residual for Tuning Vision-Language Modelscited by 123
  21. 2022Backbone is All Your Need: A Simplified Architecture for Visual Object Trackingcited by 119
  22. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  23. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 110
  24. 2022Multimodal Masked Autoencoders Learn Transferable Representationscited by 109
  25. 2022Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbonecited by 109
  26. 2022Chinese CLIP: Contrastive Vision-Language Pretraining in Chinesecited by 109
  27. 2023Scaling Laws of Synthetic Images for Model Training ... for Nowcited by 108
  28. 2022A Prompt Array Keeps the Bias Away: Debiasing Vision-Language Models with Adversarial Learningcited by 104
  29. 2022PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model Pretrainingcited by 104
  30. 2022Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIPcited by 97
  31. 2023CLIPood: Generalizing CLIP to Out-of-Distributionscited by 96
  32. 2023MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Trainingcited by 96
  33. 2023Multimodality Helps Unimodality: Cross-Modal Few-Shot Learning with Multimodal Modelscited by 91
  34. 2023Leaving Reality to Imagination: Robust Classification via Generated Datasetscited by 91
  35. 2023TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritancecited by 90
  36. 2023CLIP-KD: An Empirical Study of CLIP Model Distillationcited by 87
  37. 2023Not All Features Matter: Enhancing Few-shot CLIP with Adaptive Prior Refinementcited by 84
  38. 2023CXR-CLIP: Toward Large Scale Chest X-ray Language-Image Pre-trainingcited by 82

50 most cited of 436 All 436 (JSON)