mlsanityData to 14 Oct 2026

SVL-Adapter: Self-Supervised Adapter for Vision-Language Pretrained Models

arXiv 2210.03794 ↗First posted 7 Oct 2022cs.CVcited by 34

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Vision-language models such as CLIP are pretrained on large volumes of internet sourced image and text pairs, and have been shown to sometimes exhibit impressive zero- and low-shot image classification performance. However, due to their size, fine-tuning these models on new datasets can be prohibitively expensive, both in terms of the supervision and compute required. To combat this, a series of light-weight adaptation methods have been proposed to efficiently adapt such models when limited supervision is available. In this work, we show that while effective on internet-style datasets, even those remedies under-deliver on classification tasks with images that differ significantly from those commonly found online. To address this issue, we present a new approach called SVL-Adapter that combines the complementary strengths of both vision-language pretraining and self-supervised representation learning. We report an average classification accuracy improvement of 10% in the low-shot setting when compared to existing methods, on a set of challenging visual classification tasks. Further, we present a fully automatic way of selecting an important blending hyperparameter for our model that does not require any held-out labeled validation data. Code for our project is available here: https://github.com/omipan/svl_adapter.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 48 of its 68 references that are in the map. First posted on arXiv 7 Oct 2022.

Builds on

48 of 68 references.

Closest ancestors · 6

Same direction, newest first

  1. 2022Prompt Distribution Learning6 May
  2. 2022Unsupervised Prompt Learning for Vision-Language Models7 Apr
  3. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  4. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling6 Nov
  5. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
  6. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 42

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2020Language Models are Few-Shot Learnerscited by 32,419
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  8. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  9. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  10. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  11. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  12. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
30 more
  1. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  2. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  4. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  5. 2017Prototypical Networks for Few-shot Learningcited by 3,015
  6. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,898
  7. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  8. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  9. 2016Matching Networks for One Shot Learningcited by 2,255
  10. 2013Describing Textures in the Wildcited by 2,125
  11. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  12. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  13. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  14. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  15. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  16. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  17. 2019How Can We Know What Language Models Know?cited by 959
  18. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  19. 2018Recognition in Terra Incognitacited by 658
  20. 2018Meta-Learning for Semi-Supervised Few-Shot Classificationcited by 456
  21. 2020Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?cited by 437
  22. 2017Functional Map of the Worldcited by 338
  23. 2021With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representationscited by 326
  24. 2021Big Self-Supervised Models Advance Medical Image Classificationcited by 320
  25. 2020Geography-Aware Self-Supervised Learningcited by 190
  26. 2022Pushing the Limits of Simple Pipelines for Few-Shot Learning: External Data and Fine-Tuning Make a Differencecited by 126
  27. 2021When Does Contrastive Visual Representation Learning Work?cited by 92
  28. 2021Shallow Bayesian Meta Learning for Real-World Few-Shot Recognitioncited by 48
  29. 2022Robust and Efficient Medical Imaging with Self-Supervisioncited by 40
  30. 2021Focus on the Positives: Self-Supervised Learning for Biodiversity Monitoringcited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 11; 2024: 15; 2025: 5; 2026: 2.

34 papers.

Sort:

By area:

  1. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  2. 2023Repurposing Diffusion-Based Image Generators for Monocular Depth Estimationcited by 327
  3. 2022SuS-X: Training-Free Name-Only Transfer of Vision-Language Modelscited by 119
  4. 2023Fine-tuning can cripple your foundation model; preserving features may be the solutioncited by 96
  5. 2025Marigold: Affordable Adaptation of Diffusion-Based Image Generators for Image Analysiscited by 78
  6. 2023GraphAdapter: Tuning Vision-Language Models With Dual Knowledge Graphcited by 69
  7. 2024Improved Zero-Shot Classification by Adapting VLMs with Text Descriptionscited by 54
  8. 2023Visual Tuningcited by 34
  9. 2024Few-shot Adaptation of Multi-modal Foundation Models: A Surveycited by 28
  10. 2024Label Propagation for Zero-shot Classification with Vision-Language Modelscited by 28
  11. 2023Intra-Modal Proxy Learning for Zero-Shot Visual Categorization with CLIPcited by 27
  12. 2023Parameter-Efficient Transfer Learning for Remote Sensing Image-Text Retrievalcited by 24
22 more
  1. 2025Advances in Multimodal Adaptation and Generalization: From Traditional Approaches to Foundation Modelscited by 23
  2. 2025Bayesian Test-Time Adaptation for Vision-Language Modelscited by 22
  3. 2024Learning Cross-view Visual Geo-localization without Ground Truthcited by 18
  4. 2024Learning to Prompt Segment Anything Modelscited by 13
  5. 2023Cross-Modal Concept Learning and Inference for Vision-Language Modelscited by 12
  6. 2024Efficient Prompt Tuning of Large Vision-Language Model for Fine-Grained Ship Classificationcited by 8
  7. 2024DA-Ada: Learning Domain-Aware Adapter for Domain Adaptive Object Detectioncited by 8
  8. 2023KAFA: Rethinking Image Ad Understanding with Knowledge-Augmented Feature Adaptation of Vision-Language Modelscited by 7
  9. 2023Descriptor and Word Soups: Overcoming the Parameter Efficiency Accuracy Tradeoff for Out-of-Distribution Few-shot Learningcited by 6
  10. 2024CLIP meets DINO for Tuning Zero-Shot Classifier using Unlabeled Image Collectionscited by 5
  11. 2023UniBoost: Unsupervised Unimodal Pre-training for Boosting Zero-shot Vision-Language Taskscited by 4
  12. 2024SRL-CLIP: Efficient CLIP Video Adaptation via Structured Semantic Role Labelscited by 4
  13. 2024NODE-Adapter: Neural Ordinary Differential Equations for Better Vision-Language Reasoningcited by 3
  14. 2024Deep learning-based ecological analysis of camera trap images is impacted by training data quality and quantitycited by 3
  15. 2024Context-Aware Multimodal Pretrainingcited by 3
  16. 2024Towards Generative Class Prompt Learning for Fine-grained Visual Recognitioncited by 2
  17. 2024CLIP Adaptation by Intra-modal Overlap Reductioncited by 2
  18. 2025DesCLIP: Robust Continual Learning via General Attribute Descriptions for VLM-Based Visual Recognitioncited by 2
  19. 2026WildIng: A Wildlife Image Invariant Representation Model for Geographical Domain Shiftcited by 1
  20. 2026Hold-One-Shot-Out (HOSO) for Validation-Free Few-Shot CLIP Adapterscited by 1
  21. 2024Learning to Adapt Category Consistent Meta-Feature of CLIP for Few-Shot Classificationcited by 0
  22. 2025WEDepth: Efficient Adaptation of World Knowledge for Monocular Depth Estimationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Meta-Adapter: An Online Few-shot Learner for Vision-Language Model
  2. 2022SgVA-CLIP: Semantic-guided Visual Adapting of Vision-Language Models for Few-shot Image Classification
  3. 2024Multi-Modal Adapter for Vision-Language Models
  4. 2025CLIP-SVD: Efficient and Interpretable Vision-Language Adaptation via Singular Values
  5. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification
  6. 2023Prompt Tuning based Adapter for Vision-Language Model Adaption
  7. 2025Adapting Vision-Language Models Without Labels: A Comprehensive Survey
  8. 2023Class Incremental Learning with Pre-trained Vision-Language Models