mlsanityData to 7 Oct 2026

Contrastive Vision-Language Pre-training with Limited Resources

arXiv 2112.09331 ↗First posted 17 Dec 2021cs.CV cs.MMcited by 25

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Pioneering dual-encoder pre-training works (e.g., CLIP and ALIGN) have revealed the potential of aligning multi-modal representations with contrastive learning. However, these works require a tremendous amount of data and computational resources (e.g., billion-level web data and hundreds of GPUs), which prevent researchers with limited resources from reproduction and further exploration. To this end, we propose a stack of novel methods, which significantly cut down the heavy resource dependency and allow us to conduct dual-encoder multi-modal representation alignment with limited resources. Besides, we provide a reproducible baseline of competitive results, namely ZeroVL, with only 14M publicly accessible academic datasets and 8 V100 GPUs. Additionally, we collect 100M web data for pre-training, and achieve comparable or superior results than state-of-the-art methods, further proving the effectiveness of our methods on large-scale data. We hope that this work will provide useful data points and experience for future research in contrastive vision-language pre-training. Code is available at https://github.com/zerovl/ZeroVL.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 40 of its 60 references that are in the map. First posted on arXiv 17 Dec 2021.

Builds on

40 of 60 references.

References · 40

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  10. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  11. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  12. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
28 more
  1. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  2. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  3. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  4. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  5. 2015VQA: Visual Question Answeringcited by 2,489
  6. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  7. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  8. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  9. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  10. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  11. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  12. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  13. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  14. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  15. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  16. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  17. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  18. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  19. 2017Large Batch Training of Convolutional Networkscited by 529
  20. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  21. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  22. 2015Skip-Thought Vectorscited by 321
  23. 2019Billion-scale semi-supervised learning for image classificationcited by 232
  24. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learningcited by 187
  25. 2020Learning the Best Pooling Strategy for Visual Semantic Embeddingcited by 136
  26. 2019Augmenting Data with Mixup for Sentence Classification: An Empirical Studycited by 125
  27. 2020IMRAM: Iterative Matching with Recurrent Attention Memory for Cross-Modal Image-Text Retrievalcited by 115
  28. 2015Yin and Yang: Balancing and Answering Binary Visual Questionscited by 109

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 7; 2024: 8; 2025: 7; 2026: 2.

25 papers.

Sort:

By area:

  1. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  2. 2024Binding Touch to Everything: Learning Unified Multimodal Tactile Representationscited by 94
  3. 2023Self-Supervised Multimodal Learning: A Surveycited by 58
  4. 2023Domain Prompt Learning with Quaternion Networkscited by 17
  5. 2022Dynamic Contrastive Distillation for Image-Text Retrievalcited by 14
  6. 2024CommonIT: Commonality-Aware Instruction Tuning for Large Language Models via Data Partitionscited by 13
  7. 2023Cross-Modal Concept Learning and Inference for Vision-Language Modelscited by 12
  8. 2024TIP-I2V: A Million-Scale Real Text and Image Prompt Dataset for Image-to-Video Generationcited by 9
  9. 2025A Large-Scale Benchmark of Cross-Modal Learning for Histology and Gene Expression in Spatial Transcriptomicscited by 8
  10. 2023Vision Learners Meet Web Image-Text Pairscited by 6
  11. 2023BDC-Adapter: Brownian Distance Covariance for Better Vision-Language Reasoningcited by 6
  12. 2023Multi-Modal Representation Learning with Text-Driven Soft Maskscited by 5
13 more
  1. 2024On the Element-Wise Representation and Reasoning in Zero-Shot Image Recognition: A Systematic Surveycited by 5
  2. 2024DPA: Dual Prototypes Alignment for Unsupervised Adaptation of Vision-Language Modelscited by 5
  3. 2025Target Refocusing via Attention Redistribution for Open-Vocabulary Semantic Segmentation: An Explainability Perspectivecited by 5
  4. 2024Image Copy Detection for Diffusion Modelscited by 4
  5. 2024Nemesis: Normalizing the Soft-prompt Vectors of Vision-Language Modelscited by 2
  6. 2025SeLIP: Similarity Enhanced Contrastive Language Image Pretraining for Multi-modal Head MRIcited by 2
  7. 2024A Survey of Low-shot Vision-Language Model Adaptation via Representer Theoremcited by 1
  8. 2025Origin Identification for Text-Guided Image-to-Image Diffusion Modelscited by 1
  9. 2025Bayesian Test-time Adaptation for Object Recognition and Detection with Vision-language Modelscited by 1
  10. 2025Cross-View Open-Vocabulary Object Detection in Aerial Imagerycited by 1
  11. 2026FACTOR: Counterfactual Training-Free Test-Time Adaptation for Open-Vocabulary Object Detectioncited by 1
  12. 2025Weighted Multi-Prompt Learning with Description-free Large Language Model Distillationcited by 0
  13. 2026DIAG: Diagnostic Iterative Alignment and Generation for Data-Efficient Mathematical Preference Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Post-pre-training for Modality Alignment in Vision-Language Foundation Models
  2. 2022Vision-Language Pre-Training with Triple Contrastive Learning
  3. 2024FFF: Fixing Flawed Foundations in contrastive pre-training results in very strong Vision-Language models
  4. 2022Unifying Vision-Language Representation Space with Single-tower Transformer
  5. 2023Improved baselines for vision-language pre-training
  6. 2022Single-Stream Multi-Level Alignment for Vision-Language Pretraining
  7. 2026LeVLJEPA: End-to-End Vision-Language Pretraining Without Negatives
  8. 2023Image Captioners Are Scalable Vision Learners Too