mlsanityData to 14 Oct 2026

The Blending Ratio Is Not Where the Performance Is: Diagnosing Prototype Blending for Few-Shot Adaptation of Vision-Language Models

arXiv 2608.23634 ↗First posted 23 Aug 2026cs.CV cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

also in 1 more
Abstract (arXiv)

Many few-shot adaptation methods for vision-language models classify with a convex combination of the zero-shot text prototype and the mean of the K labelled image features, with a single blending ratio routinely tuned on held-out labels, often on the test set itself. We ask what the family's own bias-variance justification invites: what is the right ratio, can it be estimated without validation data, and is finding it where the performance is? First, the ratio minimising prototype mean-squared error has a closed form whose support-set plug-in is exactly a positive-part James-Stein coefficient shrinking towards the text prototype. Across 4,800 cells (ten datasets, five backbones including SigLIP, five shot counts, five seeds, four prompt tiers) this theoretically optimal ratio is a reliable estimate of the wrong quantity: on the 950 primary-tier cells where it is defined it trails a test-set-oracle ratio by 8.5 points. It saturates near 1, discarding the text prior for a nearest-class-mean classifier, because 78% of the text-image prototype distance it treats as bias is a class-independent offset that the arg max largely cancels. We prove the mechanism and bound its share of the damage at 26% by a counterfactual. Second, leave-one-out on the support set alone sets a ratio landing within 0.9 points of the oracle blend, so it is estimable without validation data. Third, validation-free linear probes beat even the oracle-tuned blend: CLAP by +1.9 points and LP++ by +1.5 on average, and at K >= 4 all four validation-free baselines sit above the oracle, the linear probes by margins excluding zero. These results locate the ceiling in the model class, not the hyperparameter: the ratio can be set near-optimally for free, and it is still not where the performance is. Code, cached features, per-cell records: https://huggingface.co/datasets/Liangzhi-Li/clipbench-blending

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 45 of its 63 references that are in the map. First posted on arXiv 23 Aug 2026.

Builds on

45 of 63 references.

Closest ancestors · 19

Same direction, newest first

  1. 2026Cross-Modal Prototype Alignment and Mixing for Training-Free Few-Shot Classification25 Mar
  2. 2025The Illusion of Progress? A Critical Look at Test-Time Adaptation for Vision-Language Models30 Jun
  3. 2025ProKeR: A Kernel Perspective on Few-Shot Adaptation of Large Vision-Language Models19 Jan
  4. 2024A Survey of Low-shot Vision-Language Model Adaptation via Representer Theorem15 Oct
  5. 2024AWT: Transferring Vision-Language Models via Augmentation, Weighting, and Transportation5 Jul
  6. 2024Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Models11 Apr
  7. 2024Dual Memory Networks: A Versatile Adaptation Approach for Vision-Language Models26 Mar
  8. 2023A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models20 Dec
  9. 2023Visual-Language Prompt Tuning with Knowledge-guided Context Optimization23 Mar
  10. 2023Test-Time Distribution Normalization for Contrastively Learned Vision-language Models22 Feb
  11. 2022SuS-X: Training-Free Name-Only Transfer of Vision-Language Models28 Nov
  12. 2022Task Residual for Tuning Vision-Language Models18 Nov
7 more
  1. 2022Visual Classification via Description from Large Language Models13 Oct
  2. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models15 Sep
  3. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification19 Jul
  4. 2022Prompt-aligned Gradient for Prompt Tuning30 May
  5. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  6. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
  7. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 26

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  4. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  5. 2017Prototypical Networks for Few-shot Learningcited by 3,015
  6. 2013Describing Textures in the Wildcited by 2,125
  7. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  8. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  9. 2025DINOv3cited by 1,414
  10. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,188
  11. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  12. 2021Robust fine-tuning of zero-shot modelscited by 797
14 more
  1. 2022MaPLe: Multi-modal Prompt Learningcited by 687
  2. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 561
  3. 2022What does a platypus look like? Generating customized prompts for zero-shot image classificationcited by 267
  4. 2019SimpleShot: Revisiting Nearest-Neighbor Classification for Few-Shot Learningcited by 195
  5. 2023Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learnerscited by 152
  6. 2023Multimodality Helps Unimodality: Cross-Modal Few-Shot Learning with Multimodal Modelscited by 91
  7. 2023Not All Features Matter: Enhancing Few-shot CLIP with Adaptive Prior Refinementcited by 84
  8. 2024LP++: A Surprisingly Strong Linear Probe for Few-Shot CLIPcited by 60
  9. 2024A Hard-to-Beat Baseline for Training-free CLIP-based Adaptationcited by 53
  10. 2025Few-Shot, Now for Real: Medical VLMs Adaptation without Balanced Sets or Validationcited by 7
  11. 2025Rethinking Few Shot CLIP Benchmarks: A Critical Analysis in the Inductive Settingcited by 4
  12. 2024Cross-Modal Mapping: Mitigating the Modality Gap for Few-Shot Image Classificationcited by 1
  13. 2026Benchmarking Few-shot Transferability of Pre-trained Models with Improved Evaluation Protocolscited by 1
  14. 2026Hold-One-Shot-Out (HOSO) for Validation-Free Few-Shot CLIP Adapterscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Scaling Vision-Language Models Is Not Enough to Mitigate Bias
  2. 2025Vocabulary-free few-shot learning for Vision-Language Models
  3. 2026From Inference to Adaptation: A Unified Optimal Transport View of Vision Language Model
  4. 2025Rethinking Few-Shot Adaptation of Vision-Language Models in Two Stages
  5. 2025Adapting Vision-Language Models Without Labels: A Comprehensive Survey
  6. 2025Generalizable Vision-Language Few-Shot Adaptation with Predictive Prompts and Negative Learning
  7. 2026PRiSM: Prototype Regularization for Few-Shot VLMs
  8. 2024Low-Rank Few-Shot Adaptation of Vision-Language Models