mlsanityData to 7 Oct 2026

Understanding Transferable Representation Learning and Zero-shot Transfer in CLIP

arXiv 2310.00927 ↗First posted 2 Oct 2023cs.LG cs.AI stat.MLcited by 26

Not on the map. Why?

Abstract (arXiv)

Multi-modal learning has become increasingly popular due to its ability to leverage information from different data sources (e.g., text and images) to improve the model performance. Recently, CLIP has emerged as an effective approach that employs vision-language contrastive pretraining to learn joint image and text representations and exhibits remarkable performance in zero-shot learning and text-guided natural image generation. Despite the huge practical success of CLIP, its theoretical understanding remains elusive. In this paper, we formally study transferrable representation learning underlying CLIP and demonstrate how features from different modalities get aligned. We also analyze its zero-shot transfer performance on the downstream tasks. Inspired by our analysis, we propose a new CLIP-type approach, which achieves better performance than CLIP and other state-of-the-art methods on benchmark datasets.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Builds on 42 of its 61 references that are in the map. First posted on arXiv 2 Oct 2023.

Builds on

42 of 61 references.

References · 42

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  6. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  7. 2013Describing Textures in the Wildcited by 2,110
  8. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  9. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  10. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  11. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  12. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
30 more
  1. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  2. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  3. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  4. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  5. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
  6. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  7. 2020Understanding the Behaviour of Contrastive Losscited by 397
  8. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  9. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 311
  10. 2023Multimodal Deep Learningcited by 257
  11. 2022TinyViT: Fast Pretraining Distillation for Small Vision Transformerscited by 232
  12. 2020Representation Learning via Invariant Causal Mechanismscited by 197
  13. 2022Unified Contrastive Learning in Image-Text-Label Spacecited by 194
  14. 2021Combined Scaling for Zero-shot Transfer Learningcited by 182
  15. 2020Predicting What You Already Know Helps: Provable Self-Supervised Learningcited by 152
  16. 2021What Makes Multi-modal Learning Better than Single (Provably)cited by 151
  17. 2022CyCLIP: Cyclic Contrastive Language-Image Pretrainingcited by 127
  18. 2020Learning Visual Representations with Caption Annotationscited by 123
  19. 2021Toward Understanding the Feature Learning Process of Self-supervised Contrastive Learningcited by 119
  20. 2022PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model Pretrainingcited by 104
  21. 2022Understanding Contrastive Learning Requires Incorporating Inductive Biasescited by 96
  22. 2023Factorized Contrastive Learning: Going Beyond Multi-view Redundancycited by 73
  23. 2015Predicting Deep Zero-Shot Convolutional Neural Networks using Textual Descriptionscited by 64
  24. 2020Understanding Self-supervised Learning with Dual Deep Networkscited by 62
  25. 2020Contrastive estimation reveals topic posterior information to linear modelscited by 54
  26. 2017Self-supervised learning of visual features through embedding images into text topic spacescited by 43
  27. 2023Understanding Multimodal Contrastive Learning and Incorporating Unpaired Datacited by 43
  28. 2020Self-supervised Learning from a Multi-view Perspectivecited by 18
  29. 2022Prefix Conditioning Unifies Language and Label Supervisioncited by 13
  30. 2022Generalization Bounds for Few-Shot Transfer Learning with Pretrained Classifierscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 6; 2026: 10.

26 papers.

Sort:

By area:

  1. 2024Calibrated Self-Rewarding Vision Language Modelscited by 77
  2. 2024Enhancing Large Vision Language Models with Self-Training on Image Comprehensioncited by 70
  3. 2025Temporal Preference Optimization for Long-Form Video Understandingcited by 28
  4. 2024Federated Learning from Vision-Language Foundation Models: Theoretical Analysis and Methodcited by 21
  5. 2024Fine-Grained Verifiers: Preference Modeling as Next-token Prediction in Vision-Language Alignmentcited by 17
  6. 2024Understanding Retrieval-Augmented Task Adaptation for Vision-Language Modelscited by 10
  7. 2024On the Comparison between Multi-modal and Single-modal Contrastive Learningcited by 10
  8. 2025On the Provable Importance of Gradients for Language-Assisted Image Clusteringcited by 8
  9. 2025Multi-modal contrastive learning adapts to intrinsic dimensions of shared latent variablescited by 7
  10. 2024Weighted Point Set Embedding for Multimodal Contrastive Learning Toward Optimal Similarity Metriccited by 6
  11. 2024Provably Neural Active Learning Succeeds via Prioritizing Perplexing Samplescited by 5
  12. 2026Zero-Flow Encoderscited by 4
14 more
  1. 2025FuseLIP: Multimodal Embeddings via Early Fusion of Discrete Tokenscited by 3
  2. 2024On Discriminative Probabilistic Modeling for Self-Supervised Representation Learningcited by 2
  3. 2025Theoretical Refinement of CLIP by Utilizing Linear Structure of Optimal Similaritycited by 2
  4. 2026Delving into Spectral Clustering with Vision-Language Representationscited by 2
  5. 2024Expanding Event Modality Applications through a Robust CLIP-Based Encodercited by 1
  6. 2026PrismVAU: Prompt-Refined Inference System for Multimodal Video Anomaly Understandingcited by 1
  7. 2026Debiased Negative Mining Improves Out-of-distribution Detection with Pre-trained Vision-Language Modelscited by 1
  8. 2025Understanding the Gain from Data Filtering in Multimodal Contrastive Learningcited by 0
  9. 2026Learning Subspace-Preserving Sparse Attention Graphs from Heterogeneous Multiview Datacited by 0
  10. 2026Density-Aware Translation of Spurious Correlations in Zero-Shot VLMscited by 0
  11. 2026Same Concept, Different Directions: Cross-Modal Feature Heterogeneity in Sparse Autoencoderscited by 0
  12. 2026CAST: Closed-form Analytic Semantic Transfer for Zero-Shot Classifier Extensioncited by 0
  13. 2026Expressivity In Multimodal Contrastive Learningcited by 0
  14. 2026VLM4Cluster: Benchmarking Deep Clustering In the Era of Vision-Language Pre-trainingcited by 0