# Understanding Transferable Representation Learning and Zero-shot Transfer in CLIP

arXiv: https://arxiv.org/abs/2310.00927 · HTML: https://mlsanity.com/paper/2310.00927/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 61 references that are in the map.
- First posted on arXiv 2 Oct 2023.

## Builds on (most cited first, 30 of 42)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014)
- [Contrastive Learning of Medical Visual Representations from Paired Images and Text](https://mlsanity.com/paper/2010.00747/) (2020)
- [FILIP: Fine-grained Interactive Language-Image Pre-Training](https://mlsanity.com/paper/2111.07783/) (2021)
- [LiT: Zero-Shot Transfer with Locked-image text Tuning](https://mlsanity.com/paper/2111.07991/) (2021)
- [A Theoretical Analysis of Contrastive Unsupervised Representation Learning](https://mlsanity.com/paper/1902.09229/) (2019)
- [SLIP: Self-supervision meets Language-Image Pre-training](https://mlsanity.com/paper/2112.12750/) (2021)
- [Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm](https://mlsanity.com/paper/2110.05208/) (2021)
- [Understanding the Behaviour of Contrastive Loss](https://mlsanity.com/paper/2012.09740/) (2020)
- [VirTex: Learning Visual Representations from Textual Annotations](https://mlsanity.com/paper/2006.06666/) (2020)
- [Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss](https://mlsanity.com/paper/2106.04156/) (2021)
- [Multimodal Deep Learning](https://mlsanity.com/paper/2301.04856/) (2023)
- [TinyViT: Fast Pretraining Distillation for Small Vision Transformers](https://mlsanity.com/paper/2207.10666/) (2022)
- [Representation Learning via Invariant Causal Mechanisms](https://mlsanity.com/paper/2010.07922/) (2020)
- [Unified Contrastive Learning in Image-Text-Label Space](https://mlsanity.com/paper/2204.03610/) (2022)
- [Combined Scaling for Zero-shot Transfer Learning](https://mlsanity.com/paper/2111.10050/) (2021)
- [What Makes Multi-modal Learning Better than Single (Provably)](https://mlsanity.com/paper/2106.04538/) (2021)
- [Predicting What You Already Know Helps: Provable Self-Supervised Learning](https://mlsanity.com/paper/2008.01064/) (2020)
- [CyCLIP: Cyclic Contrastive Language-Image Pretraining](https://mlsanity.com/paper/2205.14459/) (2022)
- [Learning Visual Representations with Caption Annotations](https://mlsanity.com/paper/2008.01392/) (2020)

## Built on by (most cited first, 26 of 26)

- [Calibrated Self-Rewarding Vision Language Models](https://mlsanity.com/paper/2405.14622/) (2024)
- [Enhancing Large Vision Language Models with Self-Training on Image Comprehension](https://mlsanity.com/paper/2405.19716/) (2024)
- [Temporal Preference Optimization for Long-Form Video Understanding](https://mlsanity.com/paper/2501.13919/) (2025)
- [Federated Learning from Vision-Language Foundation Models: Theoretical Analysis and Method](https://mlsanity.com/paper/2409.19610/) (2024)
- [Fine-Grained Verifiers: Preference Modeling as Next-token Prediction in Vision-Language Alignment](https://mlsanity.com/paper/2410.14148/) (2024)
- [Understanding Retrieval-Augmented Task Adaptation for Vision-Language Models](https://mlsanity.com/paper/2405.01468/) (2024)
- [On the Comparison between Multi-modal and Single-modal Contrastive Learning](https://mlsanity.com/paper/2411.02837/) (2024)
- [On the Provable Importance of Gradients for Language-Assisted Image Clustering](https://mlsanity.com/paper/2510.16335/) (2025)
- [Multi-modal contrastive learning adapts to intrinsic dimensions of shared latent variables](https://mlsanity.com/paper/2505.12473/) (2025)
- [Weighted Point Set Embedding for Multimodal Contrastive Learning Toward Optimal Similarity Metric](https://mlsanity.com/paper/2404.19228/) (2024)
- [Provably Neural Active Learning Succeeds via Prioritizing Perplexing Samples](https://mlsanity.com/paper/2406.03944/) (2024)
- [Zero-Flow Encoders](https://mlsanity.com/paper/2602.00797/) (2026)
- [FuseLIP: Multimodal Embeddings via Early Fusion of Discrete Tokens](https://mlsanity.com/paper/2506.03096/) (2025)
- [On Discriminative Probabilistic Modeling for Self-Supervised Representation Learning](https://mlsanity.com/paper/2410.09156/) (2024)
- [Theoretical Refinement of CLIP by Utilizing Linear Structure of Optimal Similarity](https://mlsanity.com/paper/2510.15508/) (2025)
- [Delving into Spectral Clustering with Vision-Language Representations](https://mlsanity.com/paper/2602.09586/) (2026)
- [Expanding Event Modality Applications through a Robust CLIP-Based Encoder](https://mlsanity.com/paper/2412.03093/) (2024)
- [PrismVAU: Prompt-Refined Inference System for Multimodal Video Anomaly Understanding](https://mlsanity.com/paper/2601.02927/) (2026)
- [Debiased Negative Mining Improves Out-of-distribution Detection with Pre-trained Vision-Language Models](https://mlsanity.com/paper/2605.23797/) (2026)
- [Understanding the Gain from Data Filtering in Multimodal Contrastive Learning](https://mlsanity.com/paper/2512.14230/) (2025)
- [Learning Subspace-Preserving Sparse Attention Graphs from Heterogeneous Multiview Data](https://mlsanity.com/paper/2605.11881/) (2026)
- [Density-Aware Translation of Spurious Correlations in Zero-Shot VLMs](https://mlsanity.com/paper/2606.01710/) (2026)
- [Same Concept, Different Directions: Cross-Modal Feature Heterogeneity in Sparse Autoencoders](https://mlsanity.com/paper/2606.29888/) (2026)
- [CAST: Closed-form Analytic Semantic Transfer for Zero-Shot Classifier Extension](https://mlsanity.com/paper/2608.13751/) (2026)
- [Expressivity In Multimodal Contrastive Learning](https://mlsanity.com/paper/2608.17203/) (2026)
- [VLM4Cluster: Benchmarking Deep Clustering In the Era of Vision-Language Pre-training](https://mlsanity.com/paper/2609.36648/) (2026)

## Abstract (arXiv)

Multi-modal learning has become increasingly popular due to its ability to leverage information from different data sources (e.g., text and images) to improve the model performance. Recently, CLIP has emerged as an effective approach that employs vision-language contrastive pretraining to learn joint image and text representations and exhibits remarkable performance in zero-shot learning and text-guided natural image generation. Despite the huge practical success of CLIP, its theoretical understanding remains elusive. In this paper, we formally study transferrable representation learning underlying CLIP and demonstrate how features from different modalities get aligned. We also analyze its zero-shot transfer performance on the downstream tasks. Inspired by our analysis, we propose a new CLIP-type approach, which achieves better performance than CLIP and other state-of-the-art methods on benchmark datasets.