mlsanityData to 7 Oct 2026

Unlocking Few-Shot Capabilities in LVLMs via Prompt Conditioning and Head Selection

arXiv 2603.24181 ↗First posted 25 Mar 2026cs.CVcited by 1

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Current Large Vision Language Models (LVLMs) excel at many zero-shot tasks like image captioning, visual question answering and OCR. However, these same models suffer from poor performance at image classification tasks, underperforming against CLIP-based methods. Notably, this gap is surprising because many LVLMs use CLIP-pretrained vision encoders. Yet LVLMs are not inherently limited by CLIP's architecture with independent vision and text encoders. In CLIP, this separation biases classification toward class-name matching rather than joint visual-text reasoning. In this paper we show that, despite their poor raw performance, LVLMs can improve visual feature class separability at inference using prompt conditioning, and LVLMs' internal representations, especially attention heads, can outperform the model itself at zero-shot and few-shot classification. We introduce Head Ensemble Classifiers (HEC) to bridge the performance gap between CLIP-based and LVLM-based classification methods. Inspired by Gaussian Discriminant Analysis, HEC ranks the most discriminative vision and text heads and combines them into a training-free classifier. We show that HEC achieves state-of-the-art performance in few-shot and zero-shot classification across 12 datasets.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 54 of its 71 references that are in the map. First posted on arXiv 25 Mar 2026.

Builds on

54 of 71 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025ProAPO: Progressively Automatic Prompt Optimization for Visual Classification27 Feb
  2. 2025ProKeR: A Kernel Perspective on Few-Shot Adaptation of Large Vision-Language Models19 Jan
  3. 2024Making Large Vision Language Models to be Good Few-shot Learners21 Aug
  4. 2023Domain Prompt Learning with Quaternion Networks12 Dec
  5. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling6 Nov

Other references · 49

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  8. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  9. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  10. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  11. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  12. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
37 more
  1. 2013Describing Textures in the Wildcited by 2,110
  2. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  3. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  4. 2025DINOv3cited by 1,297
  5. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  6. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  7. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  8. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  9. 2018Meta-learning with differentiable closed-form solverscited by 395
  10. 2024What matters when building vision-language models?cited by 321
  11. 2023In-Context Learning Creates Task Vectorscited by 302
  12. 2023Data Filtering Networkscited by 247
  13. 2025LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristicscited by 227
  14. 2024VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Taskscited by 222
  15. 2020Meta-Baseline: Exploring Simple Meta-Learning for Few-Shot Learningcited by 179
  16. 2022CALIP: Zero-Shot Enhancement of CLIP with Parameter-free Attentioncited by 103
  17. 2024Why are Visually-Grounded Language Models Bad at Image Classification?cited by 101
  18. 2024NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samplescited by 94
  19. 2023Not All Features Matter: Enhancing Few-shot CLIP with Adaptive Prior Refinementcited by 84
  20. 2023LION : Empowering Multimodal Large Language Model with Dual-Level Visual Knowledgecited by 80
  21. 2022PODA: Prompt-driven Zero-shot Domain Adaptationcited by 60
  22. 2024A Hard-to-Beat Baseline for Training-free CLIP-based Adaptationcited by 51
  23. 2020Ranking Neural Checkpointscited by 50
  24. 2024Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learningcited by 49
  25. 2023Can Multimodal Large Language Models Truly Perform Multimodal In-Context Learning?cited by 31
  26. 2025CAFe: Unifying Representation and Generation with Contrastive-Autoregressive Finetuningcited by 31
  27. 2025Analyzing and Boosting the Power of Fine-Grained Visual Recognition for Multi-modal Large Language Modelscited by 30
  28. 2024Democratizing Fine-grained Visual Recognition with Large Language Modelscited by 25
  29. 2024Finding Visual Task Vectorscited by 24
  30. 2024DocLayLLM: An Efficient Multi-modal Extension of Large Language Models for Text-rich Document Understandingcited by 23
  31. 2024VladVA: Discriminative Fine-tuning of LVLMscited by 15
  32. 2024Enhancing Few-Shot Vision-Language Classification with Large Multimodal Model Featurescited by 11
  33. 2024Envisioning Class Entity Reasoning by Large Language Models for Few-shot Learningcited by 8
  34. 2025Mimicking or Reasoning: Rethinking Multi-Modal In-Context Learning in Vision-Language Modelscited by 7
  35. 2025VT-FSL: Bridging Vision and Text with LLMs for Few-Shot Learningcited by 7
  36. 2024Text and Image Are Mutually Beneficial: Enhancing Training-Free Few-Shot Classification with CLIPcited by 6
  37. 2025What do vision-language models see in the context? Investigating multimodal in-context learningcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Are In-Context Images Worth 10 Dimensions?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026MLLMCLIP: Feature-Level Distillation of MLLM for Robust Vision-Language Representations
  2. 2025The Power of One: A Single Example is All it Takes for Segmentation in VLMs
  3. 2025Short-LVLM: Compressing and Accelerating Large Vision-Language Models by Pruning Redundant Layers
  4. 2024Bridge the Modality and Capability Gaps in Vision-Language Model Selection
  5. 2023ECO: Ensembling Context Optimization for Vision-Language Models
  6. 2025HQ-CLIP: Leveraging Large Vision-Language Models to Create High-Quality Image-Text Datasets and CLIP Models
  7. 2024Calibrating Multi-modal Representations: A Pursuit of Group Robustness without Annotations
  8. 2024Enhancing Visual Classification using Comparative Descriptors