mlsanityData to 7 Oct 2026

ProKeR: A Kernel Perspective on Few-Shot Adaptation of Large Vision-Language Models

arXiv 2501.11175 ↗First posted 19 Jan 2025cs.CV cs.AI cs.LGcited by 15

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

The growing popularity of Contrastive Language-Image Pretraining (CLIP) has led to its widespread application in various visual downstream tasks. To enhance CLIP's effectiveness and versatility, efficient few-shot adaptation techniques have been widely adopted. Among these approaches, training-free methods, particularly caching methods exemplified by Tip-Adapter, have gained attention for their lightweight adaptation without the need for additional fine-tuning. In this paper, we revisit Tip-Adapter from a kernel perspective, showing that caching methods function as local adapters and are connected to a well-established kernel literature. Drawing on this insight, we offer a theoretical understanding of how these methods operate and suggest multiple avenues for enhancing the Tip-Adapter baseline. Notably, our analysis shows the importance of incorporating global information in local adapters. Therefore, we subsequently propose a global method that learns a proximal regularizer in a reproducing kernel Hilbert space (RKHS) using CLIP as a base learner. Our method, which we call ProKeR (Proximal Kernel ridge Regression), has a closed form solution and achieves state-of-the-art performances across 11 datasets in the standard few-shot adaptation benchmark.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 46 of its 72 references that are in the map. First posted on arXiv 19 Jan 2025.

Builds on

46 of 72 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024Dual Memory Networks: A Versatile Adaptation Approach for Vision-Language Models26 Mar
  2. 2023A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models20 Dec
  3. 2023Align Your Prompts: Test-Time Prompting with Distribution Alignment for Zero-Shot Generalization2 Nov
  4. 2023Consistency-guided Prompt Learning for Vision-Language Models1 Jun
  5. 2023Black Box Few-Shot Adaptation for Vision-Language models4 Apr
  6. 2023Visual-Language Prompt Tuning with Knowledge-guided Context Optimization23 Mar
  7. 2022SuS-X: Training-Free Name-Only Transfer of Vision-Language Models28 Nov
  8. 2022Task Residual for Tuning Vision-Language Models18 Nov
  9. 2022Unified Vision and Language Prompt Learning13 Oct
  10. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models15 Sep
  11. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  12. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling6 Nov
2 more
  1. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
  2. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 32

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  4. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  5. 2013Describing Textures in the Wildcited by 2,110
  6. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  7. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  8. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  9. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  10. 2019Natural Adversarial Examplescited by 1,401
  11. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  12. 2022MaPLe: Multi-modal Prompt Learningcited by 685
20 more
  1. 2021Image Segmentation Using Text and Image Promptscited by 509
  2. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 461
  3. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  4. 2021ActionCLIP: A New Paradigm for Video Action Recognitioncited by 317
  5. 2022Frozen CLIP Models are Efficient Video Learnerscited by 181
  6. 2023CLIP2Scene: Towards Label-efficient 3D Scene Understanding by CLIPcited by 178
  7. 2023Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learnerscited by 152
  8. 2023Alpha-CLIP: A CLIP Model Focusing on Wherever You Wantcited by 150
  9. 2022Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)cited by 137
  10. 2019Improved Few-Shot Visual Classificationcited by 114
  11. 2022CALIP: Zero-Shot Enhancement of CLIP with Parameter-free Attentioncited by 103
  12. 2022Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIPcited by 97
  13. 2023Multimodality Helps Unimodality: Cross-Modal Few-Shot Learning with Multimodal Modelscited by 91
  14. 2016Orthogonal Random Featurescited by 89
  15. 2023Not All Features Matter: Enhancing Few-shot CLIP with Adaptive Prior Refinementcited by 84
  16. 2015Operator-valued Kernels for Learning from Functional Response Datacited by 59
  17. 2024A Hard-to-Beat Baseline for Training-free CLIP-based Adaptationcited by 51
  18. 2022ProposalCLIP: Unsupervised Open-Category Object Proposal Generation via Exploiting CLIP Cuescited by 42
  19. 2021Reproducing Kernel Hilbert Space, Mercer's Theorem, Eigenfunctions, Nystr"om Method, and Use of Kernels in Machine Learning: Tutorial and Surveycited by 36
  20. 2021Adversarial Regression with Doubly Non-negative Weighting Matricescited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 11.

15 papers.

Sort:

By area:

  1. 2025CLIP-SVD: Efficient and Interpretable Vision-Language Adaptation via Singular Valuescited by 3
  2. 2025Language-driven Fine-grained Retrievalcited by 2
  3. 2026Cross-Modal Prototype Alignment and Mixing for Training-Free Few-Shot Classificationcited by 2
  4. 2025UniFGVC: Universal Training-Free Few-Shot Fine-Grained Vision Classification via Attribute-Aware Multimodal Retrievalcited by 1
  5. 2026Hold-One-Shot-Out (HOSO) for Validation-Free Few-Shot CLIP Adapterscited by 1
  6. 2026Reevaluating the Intra-Modal Misalignment Hypothesis in CLIPcited by 1
  7. 2026Unlocking Few-Shot Capabilities in LVLMs via Prompt Conditioning and Head Selectioncited by 1
  8. 2025ReBaPL: Repulsive Bayesian Prompt Learningcited by 0
  9. 2026WildIng: A Wildlife Image Invariant Representation Model for Geographical Domain Shiftcited by 0
  10. 2026ReHARK: Refined Hybrid Adaptive RBF Kernels for Robust One-Shot Vision-Language Adaptationcited by 0
  11. 2026Multi-View Synergistic Learning with Vision-Language Adaption for Low-Resource Biomedical Image Classificationcited by 0
  12. 2026Decompose, Compare, and Decide: Multimodal LLMs are Implicit Few-Shot Learnerscited by 0
3 more
  1. 2026TCLA: Training-Free Class-wise Logit Adaptation for Medical Vision-Language Modelscited by 0
  2. 2026PRiSM: Prototype Regularization for Few-Shot VLMscited by 0
  3. 2026The Blending Ratio Is Not Where the Performance Is: Diagnosing Prototype Blending for Few-Shot Adaptation of Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification
  2. 2022Localized Latent Updates for Fine-Tuning Vision-Language Models
  3. 2024CLIP's Visual Embedding Projector is a Few-shot Cornucopia
  4. 2023Meta-Adapter: An Online Few-shot Learner for Vision-Language Model
  5. 2022SVL-Adapter: Self-Supervised Adapter for Vision-Language Pretrained Models
  6. 2025Attn-Adapter: Attention Is All You Need for Online Few-shot Learner of Vision-Language Model
  7. 2024A Survey of Low-shot Vision-Language Model Adaptation via Representer Theorem
  8. 2023ProbVLM: Probabilistic Adapter for Frozen Vision-Language Models