mlsanityData to 7 Oct 2026

SgVA-CLIP: Semantic-guided Visual Adapting of Vision-Language Models for Few-shot Image Classification

arXiv 2211.16191 ↗First posted 28 Nov 2022cs.CV cs.MMcited by 34

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Although significant progress has been made in few-shot learning, most of existing few-shot image classification methods require supervised pre-training on a large amount of samples of base classes, which limits their generalization ability in real world application. Recently, large-scale Vision-Language Pre-trained models (VLPs) have been gaining increasing attention in few-shot learning because they can provide a new paradigm for transferable visual representation learning with easily available text on the Web. However, the VLPs may neglect detailed visual information that is difficult to describe by language sentences, but important for learning an effective classifier to distinguish different images. To address the above problem, we propose a new framework, named Semantic-guided Visual Adapting (SgVA), which can effectively extend vision-language pre-trained models to produce discriminative adapted visual features by comprehensively using an implicit knowledge distillation, a vision-specific contrastive loss, and a cross-modal contrastive loss. The implicit knowledge distillation is designed to transfer the fine-grained cross-modal knowledge to guide the updating of the vision adapter. State-of-the-art results on 13 datasets demonstrate that the adapted visual features can well complement the cross-modal features to improve few-shot image classification.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 44 of its 64 references that are in the map. First posted on arXiv 28 Nov 2022.

Builds on

44 of 64 references.

Closest ancestors · 9

Same direction, newest first

  1. 2022Prompting through Prototype: A Prototype-based Prompt Learning on Pretrained Vision-Language Models19 Oct
  2. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification19 Jul
  3. 2022Prompt-aligned Gradient for Prompt Tuning30 May
  4. 2022Unsupervised Prompt Learning for Vision-Language Models7 Apr
  5. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  6. 2021VT-CLIP: Enhancing Vision-Language Models with Visual-guided Texts4 Dec
  7. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling6 Nov
  8. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
  9. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 35

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  8. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  9. 2016Matching Networks for One Shot Learningcited by 2,246
  10. 2013Describing Textures in the Wildcited by 2,110
  11. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  12. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
23 more
  1. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  2. 2020Meta-Learning in Neural Networks: A Surveycited by 944
  3. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  4. 2021Robust fine-tuning of zero-shot modelscited by 785
  5. 2018Meta-Learning for Semi-Supervised Few-Shot Classificationcited by 456
  6. 2014Caffe: Convolutional Architecture for Fast Feature Embeddingcited by 287
  7. 2022Pushing the Limits of Simple Pipelines for Few-Shot Learning: External Data and Fine-Tuning Make a Differencecited by 123
  8. 2022Self-Support Few-Shot Semantic Segmentationcited by 96
  9. 2019Adaptive Cross-Modal Few-Shot Learningcited by 95
  10. 2019TapNet: Neural Network Augmented with Task-Adaptive Projection for Few-Shot Learningcited by 90
  11. 2021Compressing Visual-linguistic Model via Knowledge Distillationcited by 85
  12. 2020Embedding Propagation: Smoother Manifold for Few-Shot Classificationcited by 79
  13. 2020Leveraging the Feature Distribution in Transfer-based Few-Shot Learningcited by 72
  14. 2021Partial Is Better Than All: Revisiting Fine-tuning Strategy for Few-shot Learningcited by 72
  15. 2020Multimodal Prototypical Networks for Few-shot Learningcited by 48
  16. 2019A Closer Look At Feature Space Data Augmentation For Few-Shot Intent Classificationcited by 45
  17. 2022Attribute Surrogates Learning and Spectral Tokens Pooling in Transformers for Few-shot Learningcited by 27
  18. 2021Self-supervised Contrastive Attributed Graph Clusteringcited by 18
  19. 2022EASY: Ensemble Augmented-Shot Y-shaped Learning: State-Of-The-Art Few-Shot Classification with Simple Ingredientscited by 15
  20. 2021Squeezing Backbone Feature Distributions to the Max for Efficient Few-Shot Learningcited by 14
  21. 2018Cross-Modal Distillation for RGB-Depth Person Re-Identificationcited by 13
  22. 2020Speech to Text Adaptation: Towards an Efficient Cross-Modal Distillationcited by 11
  23. 2021Attribute-Modulated Generative Meta Learning for Zero-Shot Classificationcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 15; 2025: 12; 2026: 2.

34 papers.

Sort:

By area:

  1. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  2. 2024Improved Zero-Shot Classification by Adapting VLMs with Text Descriptionscited by 52
  3. 2024Toward Visual Grounding: A Surveycited by 52
  4. 2024One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Modelscited by 48
  5. 2023CLIP-VG: Self-paced Curriculum Adapting of CLIP for Visual Groundingcited by 35
  6. 2023Visual Tuningcited by 34
  7. 2024Few-shot Adaptation of Multi-modal Foundation Models: A Surveycited by 28
  8. 2024OneRef: Unified One-tower Expression Grounding and Segmentation with Mask Referring Modelingcited by 28
  9. 2024MVP-Shot: Multi-Velocity Progressive-Alignment Framework for Few-Shot Action Recognitioncited by 25
  10. 2025Advances in Multimodal Adaptation and Generalization: From Traditional Approaches to Foundation Modelscited by 23
  11. 2023Domain Aligned CLIP for Few-shot Classificationcited by 16
  12. 2024Learning to Prompt Segment Anything Modelscited by 13
22 more
  1. 2025Logits DeConfusion with CLIP for Few-Shot Learningcited by 11
  2. 2025Proof-of-TBI -- Fine-Tuned Vision Language Model Consortium and OpenAI-o3 Reasoning LLM-Based Medical Diagnosis Support System for Mild Traumatic Brain Injury (TBI) Predictioncited by 10
  3. 2024Rethinking Misalignment in Vision-Language Model Adaptation from a Causal Perspectivecited by 9
  4. 2024RAC3: Retrieval-Augmented Corner Case Comprehension for Autonomous Driving with Vision-Language Modelscited by 7
  5. 2025Domain Generalization for Face Anti-spoofing via Content-aware Composite Prompt Engineeringcited by 7
  6. 2023Descriptor and Word Soups: Overcoming the Parameter Efficiency Accuracy Tradeoff for Out-of-Distribution Few-shot Learningcited by 6
  7. 2025Does CLIP perceive art the same way we do?cited by 2
  8. 2025An Explainable Deep Neural Network with Frequency-Aware Channel and Spatial Refinement for Flood Prediction in Sustainable Citiescited by 2
  9. 2024Class-relevant Patch Embedding Selection for Few-Shot Image Classificationcited by 1
  10. 2024AFANet: Adaptive Frequency-Aware Network for Weakly-Supervised Few-Shot Semantic Segmentationcited by 1
  11. 2024MoPD: Mixture-of-Prompts Distillation for Vision-Language Modelscited by 1
  12. 2024Cross-Modal Mapping: Mitigating the Modality Gap for Few-Shot Image Classificationcited by 1
  13. 2025UniFGVC: Universal Training-Free Few-Shot Fine-Grained Vision Classification via Attribute-Aware Multimodal Retrievalcited by 1
  14. 2025Standardization of Neuromuscular Reflex Analysis -- Role of Fine-Tuned Vision-Language Model Consortium and OpenAI gpt-oss Reasoning LLM Enabled Decision Support Systemcited by 1
  15. 2025Incorporating Scene Context and Semantic Labels for Enhanced Group-level Emotion Recognitioncited by 1
  16. 2024Using Few-Shot Learning to Classify Primary Lung Cancer and Other Malignancy with Lung Metastasis in Cytological Imaging via Endobronchial Ultrasound Procedurescited by 0
  17. 2024Enhancing Screen Time Identification in Children with a Multi-View Vision Language Model and Screen Time Trackercited by 0
  18. 2025PictSure: Pretraining Embeddings Matters for In-Context Learning Image Classifierscited by 0
  19. 2025Privacy-Shielded Image Compression: Defending Against Exploitation from Vision-Language Pretrained Modelscited by 0
  20. 2025B2N3D: Progressive Learning from Binary to N-ary Relationships for 3D Object Groundingcited by 0
  21. 2026IOTA: Corrective Knowledge-Guided Prompt Learning via Black-White Box Frameworkcited by 0
  22. 2026Structure-aware Prompt Adaptation from Seen to Unseen for Open-Vocabulary Compositional Zero-Shot Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Meta-Adapter: An Online Few-shot Learner for Vision-Language Model
  2. 2022SVL-Adapter: Self-Supervised Adapter for Vision-Language Pretrained Models
  3. 2025Auxiliary Descriptive Knowledge for Few-Shot Adaptation of Vision-Language Model
  4. 2024Low-Rank Few-Shot Adaptation of Vision-Language Models
  5. 2026Semi-Supervised Few-Shot Adaptation of Vision-Language Models
  6. 2025CLIP-SVD: Efficient and Interpretable Vision-Language Adaptation via Singular Values
  7. 2024Enhancing Fine-Grained Image Classifications via Cascaded Vision Language Models
  8. 2024Making Large Vision Language Models to be Good Few-shot Learners