mlsanityData to 7 Oct 2026

Enhancing Few-Shot Vision-Language Classification with Large Multimodal Model Features

arXiv 2412.00142 ↗First posted 28 Nov 2024cs.CV cs.AI cs.CLcited by 11

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Generative Large Multimodal Models (LMMs) like LLaVA and Qwen-VL excel at a wide variety of vision-language (VL) tasks. Despite strong performance, LMMs' generative outputs are not specialized for vision-language classification tasks (i.e., tasks with vision-language inputs and discrete labels) such as image classification and multiple-choice VQA. One key challenge in utilizing LMMs for these tasks is the extraction of useful features from generative LMMs. To overcome this, we propose an approach that leverages multimodal feature extraction from the LMM's latent space. Toward this end, we present Sparse Attention Vectors (SAVs) -- a finetuning-free method that leverages sparse attention head activations (fewer than 5% of the heads) in LMMs as strong feature representations. With only few-shot examples, SAVs demonstrate state-of-the-art performance compared to a variety of few-shot and finetuned baselines on a collection of vision-language classification tasks. Our experiments also imply that SAVs can scale in performance with additional examples and generalize to similar tasks, establishing SAVs as both effective and robust multimodal feature representations.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 79 of its 107 references that are in the map. First posted on arXiv 28 Nov 2024.

Builds on

79 of 107 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samples18 Oct
  2. 2023Compositional Chain-of-Thought Prompting for Large Multimodal Models27 Nov
  3. 2023Revisiting the Role of Language Priors in Vision-Language Models2 Jun
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan

Other references · 75

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2022Training language models to follow instructions with human feedbackcited by 15,712
  8. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  9. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  10. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  11. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  12. 2023Visual Instruction Tuningcited by 7,792
38 more
  1. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  3. 2017Neural Discrete Representation Learningcited by 4,650
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  9. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  10. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  11. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  12. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
  13. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  14. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  15. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  16. 2015VQA: Visual Question Answeringcited by 2,489
  17. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  18. 2019Towards VQA Models That Can Readcited by 1,696
  19. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  20. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  21. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  22. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  23. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  24. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  25. 2022Autoregressive Image Generation using Residual Quantizationcited by 706
  26. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  27. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  28. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
  29. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  30. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 465
  31. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 462
  32. 2021Large Dual Encoders Are Generalizable Retrieverscited by 454
  33. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  34. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  35. 2023In-Context Learning Creates Task Vectorscited by 302
  36. 2023Function Vectors in Large Language Modelscited by 292
  37. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionalitycited by 238
  38. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 171

50 most cited of 75 All 75 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 8.

11 papers.

  1. 2025A Survey on Mechanistic Interpretability for Multi-Modal Foundation Modelscited by 34
  2. 2025Granite Vision: a lightweight, open-source multimodal model for enterprise Intelligencecited by 20
  3. 2026Fine-R1: Make Multi-modal LLMs Excel in Fine-Grained Visual Recognition by Chain-of-Thought Reasoningcited by 10
  4. 2025HUMORCHAIN: Theory-Guided Multi-Stage Reasoning for Interpretable Multimodal Humor Generationcited by 4
  5. 2026SARE: Sample-wise Adaptive Reasoning for Training-free Fine-grained Visual Recognitioncited by 3
  6. 2026Building a Precise Video Language with Human-AI Oversightcited by 3
  7. 2026Unlocking Few-Shot Capabilities in LVLMs via Prompt Conditioning and Head Selectioncited by 1
  8. 2026CARPE: Context-Aware Image Representation Prioritization via Ensemble for Large Vision-Language Modelscited by 0
  9. 2026Taking Shortcuts for Categorical VQA Using Super Neuronscited by 0
  10. 2026Decompose, Compare, and Decide: Multimodal LLMs are Implicit Few-Shot Learnerscited by 0
  11. 2026AdaKerNet: Neural Kernel Decoding for Task-Adaptive Prediction with Multimodal Large Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Extraction and Analysis of Multimodal Concepts in Vision Language Models through Sparse Autoencoders
  2. 2023Pushing Boundaries: Exploring Zero Shot Object Classification with Large Multimodal Models
  3. 2026Sparse Shortcuts: Facilitating Efficient Fusion in Multimodal Large Language Models
  4. 2024Finer: Investigating and Enhancing Fine-Grained Visual Concept Recognition in Large Vision Language Models
  5. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
  6. 2025Rethinking Visual Information Processing in Multimodal LLMs
  7. 2025The Power of One: A Single Example is All it Takes for Segmentation in VLMs
  8. 2026Cascaded Sparse Autoencoders Learn Multi-Level Visual Concepts in Multimodal LLMs