# Non-Contrastive Learning Meets Language-Image Pre-Training

arXiv: https://arxiv.org/abs/2210.09304 · HTML: https://mlsanity.com/paper/2210.09304/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 74 of its 108 references that are in the map.
- First posted on arXiv 17 Oct 2022.

## Builds on (most cited first, 30 of 74)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)

## Built on by (most cited first, 25 of 25)

- [Vision-Language Models for Vision Tasks: A Survey](https://mlsanity.com/paper/2304.00685/) (2023)
- [SemiVL: Semi-Supervised Semantic Segmentation with Vision-Language Guidance](https://mlsanity.com/paper/2311.16241/) (2023)
- [S-CLIP: Semi-supervised Vision-Language Learning using Few Specialist Captions](https://mlsanity.com/paper/2305.14095/) (2023)
- [Mitigate the Gap: Investigating Approaches for Improving Cross-Modal Alignment in CLIP](https://mlsanity.com/paper/2406.17639/) (2024)
- [Improved baselines for vision-language pre-training](https://mlsanity.com/paper/2305.08675/) (2023)
- [Less is More: Removing Text-regions Improves CLIP Training Efficiency and Robustness](https://mlsanity.com/paper/2305.05095/) (2023)
- [DetailCLIP: Detail-Oriented CLIP for Fine-Grained Tasks](https://mlsanity.com/paper/2409.06809/) (2024)
- [Advancing Myopia To Holism: Fully Contrastive Language-Image Pre-training](https://mlsanity.com/paper/2412.00440/) (2024)
- [What Do You See? Enhancing Zero-Shot Image Classification with Multimodal Large Language Models](https://mlsanity.com/paper/2405.15668/) (2024)
- [DENOISER: Rethinking the Robustness for Open-Vocabulary Action Recognition](https://mlsanity.com/paper/2404.14890/) (2024)
- [MOFI: Learning Image Representations from Noisy Entity Annotated Images](https://mlsanity.com/paper/2306.07952/) (2023)
- [Evolution-based Region Adversarial Prompt Learning for Robustness Enhancement in Vision-Language Models](https://mlsanity.com/paper/2503.12874/) (2025)
- [SemiDAViL: Semi-supervised Domain Adaptation with Vision-Language Guidance for Semantic Segmentation](https://mlsanity.com/paper/2504.06389/) (2025)
- [ToxicTextCLIP: Text-Based Poisoning and Backdoor Attacks on CLIP Pre-training](https://mlsanity.com/paper/2511.00446/) (2025)
- [PC2: Pseudo-Classification Based Pseudo-Captioning for Noisy Correspondence Learning in Cross-Modal Retrieval](https://mlsanity.com/paper/2408.01349/) (2024)
- [Explaining Caption-Image Interactions in CLIP Models with Second-Order Attributions](https://mlsanity.com/paper/2408.14153/) (2024)
- [Cross-Modal Distillation For Widely Differing Modalities](https://mlsanity.com/paper/2507.16296/) (2025)
- [Bridging Visual Affective Gap: Borrowing Textual Knowledge by Learning from Noisy Image-Text Pairs](https://mlsanity.com/paper/2511.17103/) (2025)
- [Exploiting Supervised Poison Vulnerability to Strengthen Self-Supervised Defense](https://mlsanity.com/paper/2409.08509/) (2024)
- [SigCLR: Sigmoid Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2410.17427/) (2024)
- [Lost in Translation? Vocabulary Alignment for Source-Free Adaptation in Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2509.15225/) (2025)
- [Mining Open Semantics from CLIP: A Relation Transition Perspective for Few-Shot Learning](https://mlsanity.com/paper/2406.11252/) (2024)
- [Uncovering RL Integration in SSL Loss: Objective-Specific Implications for Data-Efficient RL](https://mlsanity.com/paper/2410.17428/) (2024)
- [Weighted Multi-Prompt Learning with Description-free Large Language Model Distillation](https://mlsanity.com/paper/2507.07147/) (2025)
- [CLIPin: A Non-contrastive Plug-in to CLIP for Multimodal Semantic Alignment](https://mlsanity.com/paper/2508.06434/) (2025)

## Abstract (arXiv)

Contrastive language-image pre-training (CLIP) serves as a de-facto standard to align images and texts. Nonetheless, the loose correlation between images and texts of web-crawled data renders the contrastive objective data inefficient and craving for a large training batch size. In this work, we explore the validity of non-contrastive language-image pre-training (nCLIP), and study whether nice properties exhibited in visual self-supervised models can emerge. We empirically observe that the non-contrastive objective nourishes representation learning while sufficiently underperforming under zero-shot recognition. Based on the above study, we further introduce xCLIP, a multi-tasking framework combining CLIP and nCLIP, and show that nCLIP aids CLIP in enhancing feature semantics. The synergy between two objectives lets xCLIP enjoy the best of both worlds: superior performance in both zero-shot transfer and representation learning. Systematic evaluation is conducted spanning a wide variety of downstream tasks including zero-shot classification, out-of-domain classification, retrieval, visual representation learning, and textual representation learning, showcasing a consistent performance gain and validating the effectiveness of xCLIP.