mlsanityData to 7 Oct 2026

Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm

arXiv 2110.05208 ↗First posted 11 Oct 2021cs.CVcited by 405

Not on the map. Why?

Cited by 153 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recently, large-scale Contrastive Language-Image Pre-training (CLIP) has attracted unprecedented attention for its impressive zero-shot recognition ability and excellent transferability to downstream tasks. However, CLIP is quite data-hungry and requires 400M image-text pairs for pre-training, thereby restricting its adoption. This work proposes a novel training paradigm, Data efficient CLIP (DeCLIP), to alleviate this limitation. We demonstrate that by carefully utilizing the widespread supervision among the image-text pairs, our De-CLIP can learn generic visual features more efficiently. Instead of using the single image-text contrastive supervision, we fully exploit data potential through the use of (1) self-supervision within each modality; (2) multi-view supervision across modalities; (3) nearest-neighbor supervision from other similar pairs. Benefiting from intrinsic supervision, our DeCLIP-ResNet50 can achieve 60.4% zero-shot top1 accuracy on ImageNet, which is 0.8% above the CLIP-ResNet50 while using 7.1 x fewer data. Our DeCLIP-ResNet50 outperforms its counterpart in 8 out of 11 visual datasets when transferred to downstream tasks. Moreover, Scaling up the model and computing also works well in our framework.Our code, dataset and models are released at: https://github.com/Sense-GVT/DeCLIP

Key facts. Cited by 405 papers in the map (data through 7 Oct 2026). Builds on 47 of its 61 references that are in the map. First posted on arXiv 11 Oct 2021.

Builds on

47 of 61 references.

References · 47

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  8. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  9. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  10. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  11. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  12. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
35 more
  1. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  2. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  3. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  4. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  5. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  6. 2015VQA: Visual Question Answeringcited by 2,489
  7. 2015Learning Deep Features for Discriminative Localizationcited by 2,430
  8. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  9. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  10. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,160
  11. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  12. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  13. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  14. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  15. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
  16. 2020Designing Network Design Spacescited by 954
  17. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  18. 2019EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Taskscited by 800
  19. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  20. 2018Do Better ImageNet Models Transfer Better?cited by 560
  21. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  22. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  23. 2021With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representationscited by 324
  24. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  25. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  26. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  27. 2021Self-supervised Pretraining of Visual Features in the Wildcited by 197
  28. 2015Learning Visual Features from Large Weakly Supervised Datacited by 165
  29. 2020Learning Visual Representations with Caption Annotationscited by 123
  30. 2021Multimodal Contrastive Training for Visual Representation Learningcited by 122
  31. 2018Dynamic Fusion with Intra- and Inter- Modality Attention Flow for Visual Question Answeringcited by 113
  32. 2021WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Trainingcited by 84
  33. 2021Revisiting Contrastive Methods for Unsupervised Learning of Visual Representationscited by 48
  34. 2017Self-supervised learning of visual features through embedding images into text topic spacescited by 43
  35. 2021Data-Efficient Language-Supervised Zero-Shot Learning with Self-Distillationcited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 75; 2023: 133; 2024: 103; 2025: 66; 2026: 23.

405 papers.

Sort:

By area:

  1. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  2. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  3. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  4. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  5. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 564
  6. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 560
  7. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
  8. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 429
  9. 2022Multimodal Learning with Transformers: A Surveycited by 399
  10. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Modelscited by 371
  11. 2023RemoteCLIP: A Vision Language Foundation Model for Remote Sensingcited by 357
  12. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classificationcited by 355
38 more
  1. 2022Making the Most of Text Semantics to Improve Biomedical Vision--Language Processingcited by 336
  2. 2024Long-CLIP: Unlocking the Long-Text Capability of CLIPcited by 298
  3. 2023Improving CLIP Training with Language Rewritescited by 216
  4. 2022Delving into Out-of-Distribution Detection with Vision-Language Representationscited by 215
  5. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 215
  6. 2022MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretrainingcited by 187
  7. 2022A Survey of Vision-Language Pre-Trained Modelscited by 177
  8. 2024An Introduction to Vision-Language Modelingcited by 175
  9. 2022SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentationcited by 162
  10. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  11. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  12. 2023RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote Sensingcited by 150
  13. 2022ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Modelscited by 141
  14. 2022PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models -- Federated Learning in Age of Foundation Modelcited by 133
  15. 2023Weakly Supervised 3D Open-vocabulary Segmentationcited by 133
  16. 2023Hyperbolic Image-Text Representationscited by 132
  17. 2023Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Imagescited by 132
  18. 2022A Molecular Multimodal Foundation Model Associating Molecule Graphs with Natural Languagecited by 129
  19. 2024Exploring scalable medical image encoders beyond text supervisioncited by 129
  20. 2023MolCA: Molecular Graph-Language Modeling with Cross-Modal Projector and Uni-Modal Adaptercited by 128
  21. 2022X-Pool: Cross-Modal Language-Video Attention for Text-Video Retrievalcited by 127
  22. 2022CyCLIP: Cyclic Contrastive Language-Image Pretrainingcited by 127
  23. 2022Task Residual for Tuning Vision-Language Modelscited by 122
  24. 2022SuS-X: Training-Free Name-Only Transfer of Vision-Language Modelscited by 118
  25. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  26. 2023BadCLIP: Dual-Embedding Guided Backdoor Attack on Multimodal Contrastive Learningcited by 112
  27. 2022Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbonecited by 109
  28. 2023Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlookcited by 109
  29. 2022Chinese CLIP: Contrastive Vision-Language Pretraining in Chinesecited by 108
  30. 2022Multimodal Masked Autoencoders Learn Transferable Representationscited by 107
  31. 2022PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model Pretrainingcited by 104
  32. 2022PromptCap: Prompt-Guided Task-Aware Image Captioningcited by 104
  33. 2022Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIPcited by 97
  34. 2023CLIPood: Generalizing CLIP to Out-of-Distributionscited by 96
  35. 2023MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Trainingcited by 96
  36. 2023Vision-Language Models in Remote Sensing: Current Progress and Future Trendscited by 93
  37. 2023Multimodality Helps Unimodality: Cross-Modal Few-Shot Learning with Multimodal Modelscited by 91
  38. 2023Leaving Reality to Imagination: Robust Classification via Generated Datasetscited by 91

50 most cited of 405 All 405 (JSON)