mlsanityData to 7 Oct 2026

Intra-Modal Proxy Learning for Zero-Shot Visual Categorization with CLIP

arXiv 2310.19752 ↗First posted 30 Oct 2023cs.CV cs.LGcited by 27

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Vision-language pre-training methods, e.g., CLIP, demonstrate an impressive zero-shot performance on visual categorizations with the class proxy from the text embedding of the class name. However, the modality gap between the text and vision space can result in a sub-optimal performance. We theoretically show that the gap cannot be reduced sufficiently by minimizing the contrastive loss in CLIP and the optimal proxy for vision tasks may reside only in the vision space. Therefore, given unlabeled target vision data, we propose to learn the vision proxy directly with the help from the text proxy for zero-shot transfer. Moreover, according to our theoretical analysis, strategies are developed to further refine the pseudo label obtained by the text proxy to facilitate the intra-modal proxy learning (InMaP) for vision. Experiments on extensive downstream tasks confirm the effectiveness and efficiency of our proposal. Concretely, InMaP can obtain the vision proxy within one minute on a single GPU while improving the zero-shot accuracy from 77.02% to 80.21% on ImageNet with ViT-L/14@336 pre-trained by CLIP. Code is available at urlhttps://github.com/idstcv/InMaP.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 30 of its 41 references that are in the map. First posted on arXiv 30 Oct 2023.

Builds on

30 of 41 references.

Closest ancestors · 8

Same direction, newest first

  1. 2022SuS-X: Training-Free Name-Only Transfer of Vision-Language Models28 Nov
  2. 2022Visual Classification via Description from Large Language Models13 Oct
  3. 2022SVL-Adapter: Self-Supervised Adapter for Vision-Language Pretrained Models7 Oct
  4. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models15 Sep
  5. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification19 Jul
  6. 2022Unsupervised Prompt Learning for Vision-Language Models7 Apr
  7. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
  8. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 22

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  8. 2021Convex optimizationcited by 2,479
  9. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,277
  10. 2013Describing Textures in the Wildcited by 2,110
  11. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,080
  12. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
10 more
  1. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  2. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  3. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  4. 2017Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Uglycited by 536
  5. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  6. 2015Label-Embedding for Image Classificationcited by 164
  7. 2019SoftTriple Loss: Deep Metric Learning Without Triplet Samplingcited by 163
  8. 2021HSVA: Hierarchical Semantic-Visual Adaptation for Zero-Shot Learningcited by 57
  9. 2021TransZero++: Cross Attribute-Guided Transformer for Zero-Shot Learningcited by 28
  10. 2023Improved Visual Fine-tuning with Natural Language Supervisioncited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 12; 2026: 6.

27 papers.

Sort:

By area:

  1. 2025MMTok: Multimodal Coverage Maximization for Efficient Inference of VLMscited by 29
  2. 2024Label Propagation for Zero-shot Classification with Vision-Language Modelscited by 28
  3. 2024Enhancing Zero-Shot Vision Models by Label-Free Prompt Distribution Learning and Bias Correctingcited by 26
  4. 2024Wings: Learning Multimodal LLMs without Text-only Forgettingcited by 23
  5. 2024Online Zero-Shot Classification with CLIPcited by 22
  6. 2024Multi-Modal Proxy Learning Towards Personalized Visual Multiple Clusteringcited by 21
  7. 2025Post-pre-training for Modality Alignment in Vision-Language Foundation Modelscited by 18
  8. 2025Logits DeConfusion with CLIP for Few-Shot Learningcited by 11
  9. 2024Building a Multi-modal Spatiotemporal Expert for Zero-shot Action Recognition with CLIPcited by 9
  10. 2025Multi-modal Reference Learning for Fine-grained Text-to-Image Retrievalcited by 6
  11. 2024On the Element-Wise Representation and Reasoning in Zero-Shot Image Recognition: A Systematic Surveycited by 5
  12. 2024DPA: Dual Prototypes Alignment for Unsupervised Adaptation of Vision-Language Modelscited by 5
15 more
  1. 2025Learning Sparsity for Effective and Efficient Music Performance Question Answeringcited by 4
  2. 2024LatteCLIP: Unsupervised CLIP Fine-Tuning via LMM-Synthetic Textscited by 3
  3. 2025Q-CLIP: Unleashing the Power of Vision-Language Models for Video Quality Assessment through Unified Cross-Modal Adaptationcited by 3
  4. 2025Learning Visual Proxy for Compositional Zero-Shot Learningcited by 2
  5. 2025RANa: Retrieval-Augmented Navigationcited by 2
  6. 2025TeDA: Boosting Vision-Lanuage Models for Zero-Shot 3D Object Retrieval via Testing-time Distribution Alignmentcited by 2
  7. 2025Selective, Controlled and Domain-Agnostic Unlearning in Pretrained CLIP: A Training- and Data-Free Approachcited by 2
  8. 2026CARPRT: Class-Aware Zero-Shot Prompt Reweighting for Black-Box Vision-Language Modelscited by 2
  9. 2025Uniformity First: Uniformity-aware Test-time Adaptation of Vision-language Models against Image Corruptioncited by 1
  10. 2026Reevaluating the Intra-Modal Misalignment Hypothesis in CLIPcited by 1
  11. 2025GRIT: Graph-Regularized Logit Refinement for Zero-shot Cell Type Annotationcited by 0
  12. 2026IGLOSS: Image Generation for Lidar Open-vocabulary Semantic Segmentationcited by 0
  13. 2026Label Shift Aware Adaptation for Online Zero-shot Learning with Contrastive Language-Image Pre-Training (CLIP)cited by 0
  14. 2026Same Concept, Different Directions: Cross-Modal Feature Heterogeneity in Sparse Autoencoderscited by 0
  15. 2026Visual Distribution Anchoring for Efficient Prompt Tuningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Zoom-shot: Fast and Efficient Unsupervised Zero-Shot Transfer of CLIP to Vision Encoders with Multimodal Loss
  2. 2026Will It Zero-Shot?: Predicting Zero-Shot Classification Performance For Arbitrary Queries
  3. 2022Zero-shot Image Captioning by Anchor-augmented Vision-Language Space Alignment
  4. 2024BaFTA: Backprop-Free Test-Time Adaptation For Zero-Shot Vision-Language Models
  5. 2023UniFine: A Unified and Fine-grained Approach for Zero-shot Vision-Language Understanding
  6. 2024Grounding Descriptions in Images informs Zero-Shot Visual Recognition
  7. 2023Filtering, Distillation, and Hard Negatives for Vision-Language Pre-Training
  8. 2026Explaining CLIP Zero-shot Predictions Through Concepts