mlsanityData to 14 Oct 2026

Exploiting Category Names for Few-Shot Classification with Vision-Language Models

arXiv 2211.16594 ↗First posted 29 Nov 2022cs.CVcited by 3

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Vision-language foundation models pretrained on large-scale data provide a powerful tool for many visual understanding tasks. Notably, many vision-language models build two encoders (visual and textual) that can map two modalities into the same embedding space. As a result, the learned representations achieve good zero-shot performance on tasks like image classification. However, when there are only a few examples per category, the potential of large vision-language models is often underperformed, mainly due to the gap between a large number of parameters and a relatively small amount of training data. This paper shows that we can significantly improve the performance of few-shot classification by using the category names to initialize the classification head. With the proposed category name initialization method, our model obtains the state-of-the-art performance on a number of few-shot image classification benchmarks (e.g., 87.37% on ImageNet and 96.08% on Stanford Cars, both using five-shot learning).

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 32 of its 42 references that are in the map. First posted on arXiv 29 Nov 2022.

Builds on

32 of 42 references.

Closest ancestors · 5

Same direction, newest first

  1. 2022SuS-X: Training-Free Name-Only Transfer of Vision-Language Models28 Nov
  2. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  3. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling6 Nov
  4. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
  5. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 27

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  3. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  6. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  7. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  8. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  9. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  10. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  11. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  12. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
15 more
  1. 2019Natural Adversarial Examplescited by 1,408
  2. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  3. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  4. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  5. 2014Show and Tell: A Neural Image Caption Generatorcited by 879
  6. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  7. 2021Robust fine-tuning of zero-shot modelscited by 797
  8. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
  9. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 647
  10. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
  11. 2014Long-term Recurrent Convolutional Networks for Visual Recognition and Descriptioncited by 484
  12. 2022Context Autoencoder for Self-Supervised Representation Learningcited by 297
  13. 2016What makes ImageNet good for transfer learning?cited by 192
  14. 2022ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Modelscited by 141
  15. 2019Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modelingcited by 86

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 1.

3 papers.

  1. 2023Less is More: Removing Text-regions Improves CLIP Training Efficiency and Robustnesscited by 19
  2. 2023Domain Aligned CLIP for Few-shot Classificationcited by 16
  3. 2024Review of Zero-Shot and Few-Shot AI Algorithms in The Medical Domaincited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2024Enhancing Visual Classification using Comparative Descriptors
  2. 2024Making Large Vision Language Models to be Good Few-shot Learners
  3. 2024LLM meets Vision-Language Models for Zero-Shot One-Class Classification
  4. 2023Vocabulary-free Image Classification
  5. 2024Envisioning Class Entity Reasoning by Large Language Models for Few-shot Learning
  6. 2024Few-Shot Recognition via Stage-Wise Retrieval-Augmented Finetuning
  7. 2022Visual Classification via Description from Large Language Models
  8. 2024Retrieval-enriched zero-shot image classification in low-resource domains