mlsanityData to 14 Oct 2026

CLIP's Visual Embedding Projector is a Few-shot Cornucopia

arXiv 2410.05270 ↗First posted 7 Oct 2024cs.CVcited by 1

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

We introduce ProLIP, a simple and architecture-agnostic method for adapting contrastively pretrained vision-language models, such as CLIP, to few-shot classification. ProLIP fine-tunes the vision encoder's projection matrix with Frobenius norm regularization on its deviation from the pretrained weights. It achieves state-of-the-art performance on 11 few-shot classification benchmarks under both ``few-shot validation'' and ``validation-free'' settings. Moreover, by rethinking the non-linear CLIP-Adapter through ProLIP's lens, we design a Regularized Linear Adapter (RLA) that performs better, requires no hyperparameter tuning, is less sensitive to learning rate values, and offers an alternative to ProLIP in black-box scenarios where model weights are inaccessible. Beyond few-shot classification, ProLIP excels in cross-dataset transfer, domain generalization, base-to-new class generalization, and test-time adaptation--where it outperforms prompt tuning while being an order of magnitude faster to train. Code is available at https://github.com/astra-vision/ProLIP .

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 53 of its 71 references that are in the map. First posted on arXiv 7 Oct 2024.

Builds on

53 of 71 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024Low-Rank Few-Shot Adaptation of Vision-Language Models28 May
  2. 2024AMU-Tuning: Effective Logit Bias for CLIP-based Few-shot Learning13 Apr
  3. 2023A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models20 Dec
  4. 2023Meta-Adapter: An Online Few-shot Learner for Vision-Language Model7 Nov
  5. 2023Visual-Language Prompt Tuning with Knowledge-guided Context Optimization23 Mar
  6. 2022Task Residual for Tuning Vision-Language Models18 Nov
  7. 2022Visual Classification via Description from Large Language Models13 Oct
  8. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models15 Sep
  9. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification19 Jul
  10. 2022Prompt-aligned Gradient for Prompt Tuning30 May
  11. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  12. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
1 more
  1. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 40

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  9. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  10. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  11. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  12. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
28 more
  1. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  2. 2013Describing Textures in the Wildcited by 2,125
  3. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  4. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  5. 2022Visual Prompt Tuningcited by 1,681
  6. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  7. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  8. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  9. 2019Natural Adversarial Examplescited by 1,408
  10. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  11. 2021Grounded Language-Image Pre-trainingcited by 1,156
  12. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  13. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 739
  14. 2022MaPLe: Multi-modal Prompt Learningcited by 687
  15. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
  16. 2023Self-regulating Prompts: Foundational Model Adaptation without Forgettingcited by 272
  17. 2020AdapterDrop: On the Efficiency of Adapters in Transformerscited by 224
  18. 2022PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learningcited by 187
  19. 2023Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learnerscited by 152
  20. 2023CLIPood: Generalizing CLIP to Out-of-Distributionscited by 96
  21. 2023Multimodality Helps Unimodality: Cross-Modal Few-Shot Learning with Multimodal Modelscited by 91
  22. 2023Not All Features Matter: Enhancing Few-shot CLIP with Adaptive Prior Refinementcited by 84
  23. 2023Stronger, Fewer, & Superior: Harnessing Vision Foundation Models for Domain Generalized Semantic Segmentationcited by 83
  24. 2023Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuningcited by 64
  25. 2024LP++: A Surprisingly Strong Linear Probe for Few-Shot CLIPcited by 60
  26. 2024A Hard-to-Beat Baseline for Training-free CLIP-based Adaptationcited by 53
  27. 2020Distance-Based Regularisation of Deep Networks for Fine-Tuningcited by 48
  28. 2023Domain Aligned CLIP for Few-shot Classificationcited by 16

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024Domain Adaptation with a Single Vision-Language Embeddingcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling
  2. 2025ProKeR: A Kernel Perspective on Few-Shot Adaptation of Large Vision-Language Models
  3. 2023ReCLIP: Refine Contrastive Language Image Pre-Training with Source Free Domain Adaptation
  4. 2025Rethinking Few-Shot Adaptation of Vision-Language Models in Two Stages
  5. 2025CLIP-SVD: Efficient and Interpretable Vision-Language Adaptation via Singular Values
  6. 2026Koo-Fu CLIP: Closed-Form Adaptation of Vision-Language Models via Fukunaga-Koontz Linear Discriminant Analysis
  7. 2024BaFTA: Backprop-Free Test-Time Adaptation For Zero-Shot Vision-Language Models
  8. 2022Localized Latent Updates for Fine-Tuning Vision-Language Models