mlsanityData to 7 Oct 2026

Non-Contrastive Learning Meets Language-Image Pre-Training

arXiv 2210.09304 ↗First posted 17 Oct 2022cs.CVcited by 25

Not on the map. Why?

Abstract (arXiv)

Contrastive language-image pre-training (CLIP) serves as a de-facto standard to align images and texts. Nonetheless, the loose correlation between images and texts of web-crawled data renders the contrastive objective data inefficient and craving for a large training batch size. In this work, we explore the validity of non-contrastive language-image pre-training (nCLIP), and study whether nice properties exhibited in visual self-supervised models can emerge. We empirically observe that the non-contrastive objective nourishes representation learning while sufficiently underperforming under zero-shot recognition. Based on the above study, we further introduce xCLIP, a multi-tasking framework combining CLIP and nCLIP, and show that nCLIP aids CLIP in enhancing feature semantics. The synergy between two objectives lets xCLIP enjoy the best of both worlds: superior performance in both zero-shot transfer and representation learning. Systematic evaluation is conducted spanning a wide variety of downstream tasks including zero-shot classification, out-of-domain classification, retrieval, visual representation learning, and textual representation learning, showcasing a consistent performance gain and validating the effectiveness of xCLIP.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 74 of its 108 references that are in the map. First posted on arXiv 17 Oct 2022.

Builds on

74 of 108 references.

References · 74

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  8. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  9. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  10. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  11. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  12. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
38 more
  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  4. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  5. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  6. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  7. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  8. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  9. 2013Describing Textures in the Wildcited by 2,110
  10. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  11. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  12. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  13. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  14. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  15. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  16. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  17. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  18. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  19. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  20. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  21. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
  22. 2019Natural Adversarial Examplescited by 1,401
  23. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  24. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  25. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  26. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  27. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  28. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  29. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  30. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  31. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  32. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  33. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  34. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  35. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  36. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 791
  37. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  38. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645

50 most cited of 74 All 74 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 6; 2024: 11; 2025: 8.

25 papers.

Sort:

By area:

  1. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  2. 2023SemiVL: Semi-Supervised Semantic Segmentation with Vision-Language Guidancecited by 41
  3. 2023S-CLIP: Semi-supervised Vision-Language Learning using Few Specialist Captionscited by 29
  4. 2024Mitigate the Gap: Investigating Approaches for Improving Cross-Modal Alignment in CLIPcited by 23
  5. 2023Improved baselines for vision-language pre-trainingcited by 20
  6. 2023Less is More: Removing Text-regions Improves CLIP Training Efficiency and Robustnesscited by 19
  7. 2024DetailCLIP: Detail-Oriented CLIP for Fine-Grained Taskscited by 15
  8. 2024Advancing Myopia To Holism: Fully Contrastive Language-Image Pre-trainingcited by 13
  9. 2024What Do You See? Enhancing Zero-Shot Image Classification with Multimodal Large Language Modelscited by 10
  10. 2024DENOISER: Rethinking the Robustness for Open-Vocabulary Action Recognitioncited by 8
  11. 2023MOFI: Learning Image Representations from Noisy Entity Annotated Imagescited by 7
  12. 2025Evolution-based Region Adversarial Prompt Learning for Robustness Enhancement in Vision-Language Modelscited by 6
13 more
  1. 2024PC2: Pseudo-Classification Based Pseudo-Captioning for Noisy Correspondence Learning in Cross-Modal Retrievalcited by 3
  2. 2024Explaining Caption-Image Interactions in CLIP Models with Second-Order Attributionscited by 3
  3. 2025SemiDAViL: Semi-supervised Domain Adaptation with Vision-Language Guidance for Semantic Segmentationcited by 3
  4. 2025ToxicTextCLIP: Text-Based Poisoning and Backdoor Attacks on CLIP Pre-trainingcited by 3
  5. 2025Cross-Modal Distillation For Widely Differing Modalitiescited by 2
  6. 2025Bridging Visual Affective Gap: Borrowing Textual Knowledge by Learning from Noisy Image-Text Pairscited by 2
  7. 2024Exploiting Supervised Poison Vulnerability to Strengthen Self-Supervised Defensecited by 1
  8. 2024SigCLR: Sigmoid Contrastive Learning of Visual Representationscited by 1
  9. 2025Lost in Translation? Vocabulary Alignment for Source-Free Adaptation in Open-Vocabulary Semantic Segmentationcited by 1
  10. 2024Mining Open Semantics from CLIP: A Relation Transition Perspective for Few-Shot Learningcited by 0
  11. 2024Uncovering RL Integration in SSL Loss: Objective-Specific Implications for Data-Efficient RLcited by 0
  12. 2025Weighted Multi-Prompt Learning with Description-free Large Language Model Distillationcited by 0
  13. 2025CLIPin: A Non-contrastive Plug-in to CLIP for Multimodal Semantic Alignmentcited by 0