mlsanityData to 7 Oct 2026

SuS-X: Training-Free Name-Only Transfer of Vision-Language Models

arXiv 2211.16198 ↗First posted 28 Nov 2022cs.CV cs.CL cs.MMcited by 118

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Contrastive Language-Image Pre-training (CLIP) has emerged as a simple yet effective way to train large-scale vision-language models. CLIP demonstrates impressive zero-shot classification and retrieval on diverse downstream tasks. However, to leverage its full potential, fine-tuning still appears to be necessary. Fine-tuning the entire CLIP model can be resource-intensive and unstable. Moreover, recent methods that aim to circumvent this need for fine-tuning still require access to images from the target distribution. In this paper, we pursue a different approach and explore the regime of training-free "name-only transfer" in which the only knowledge we possess about the downstream task comprises the names of downstream target categories. We propose a novel method, SuS-X, consisting of two key building blocks -- SuS and TIP-X, that requires neither intensive fine-tuning nor costly labelled data. SuS-X achieves state-of-the-art zero-shot classification results on 19 benchmark datasets. We further show the utility of TIP-X in the training-free few-shot setting, where we again achieve state-of-the-art results over strong training-free baselines. Code is available at https://github.com/vishaal27/SuS-X.

Key facts. Cited by 118 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 62 of its 95 references that are in the map. First posted on arXiv 28 Nov 2022.

Builds on

62 of 95 references.

Closest ancestors · 14

Same direction, newest first

  1. 2022Visual Classification via Description from Large Language Models13 Oct
  2. 2022SVL-Adapter: Self-Supervised Adapter for Vision-Language Pretrained Models7 Oct
  3. 2022PLOT: Prompt Learning with Optimal Transport for Vision-Language Models3 Oct
  4. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models15 Sep
  5. 2022Prompt Tuning with Soft Context Sharing for Vision-Language Models29 Aug
  6. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification19 Jul
  7. 2022Prompt Distribution Learning6 May
  8. 2022PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model Pretraining29 Apr
  9. 2022Unsupervised Prompt Learning for Vision-Language Models7 Apr
  10. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  11. 2021VT-CLIP: Enhancing Vision-Language Models with Visual-guided Texts4 Dec
  12. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling6 Nov
2 more
  1. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
  2. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 48

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  7. 2022Classifier-Free Diffusion Guidancecited by 5,717
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  10. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  11. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  12. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
36 more
  1. 2013Describing Textures in the Wildcited by 2,110
  2. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  3. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  4. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  5. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  6. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  7. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  8. 2018Recognition in Terra Incognitacited by 655
  9. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  10. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  11. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  12. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  13. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  14. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  15. 2017Functional Map of the Worldcited by 338
  16. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  17. 2022Is synthetic data from generative models ready for image recognition?cited by 320
  18. 2022Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Modelcited by 317
  19. 2022X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrievalcited by 279
  20. 2022What does a platypus look like? Generating customized prompts for zero-shot image classificationcited by 263
  21. 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212
  22. 2022Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)cited by 137
  23. 2022CyCLIP: Cyclic Contrastive Language-Image Pretrainingcited by 127
  24. 2022ReCo: Retrieve and Co-segment for Zero-shot Transfercited by 115
  25. 2022CALIP: Zero-Shot Enhancement of CLIP with Parameter-free Attentioncited by 103
  26. 2022Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIPcited by 97
  27. 2022Continual Pre-Training Mitigates Forgetting in Language and Visioncited by 82
  28. 2022CLiMB: A Continual Learning Benchmark for Vision-and-Language Taskscited by 68
  29. 2022A CLIP-Hitchhiker's Guide to Long Video Retrievalcited by 57
  30. 2022Is a Caption Worth a Thousand Images? A Controlled Study for Representation Learningcited by 39
  31. 2022Learning Visual Representation from Modality-Shared Contrastive Language-Image Pre-trainingcited by 32
  32. 2022Multimodal Knowledge Alignment with Reinforcement Learningcited by 30
  33. 2021Embedding Arithmetic of Multimodal Queries for Image Retrievalcited by 21
  34. 2022NamedMask: Distilling Segmenters from Complementary Foundation Modelscited by 20
  35. 2020COBRA: Contrastive Bi-Modal Representation Algorithmcited by 12
  36. 2022Caption supervision enables robust learnerscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 26; 2024: 44; 2025: 32; 2026: 15.

118 papers.

Sort:

By area:

  1. 2023A Survey on Multimodal Large Language Modelscited by 898
  2. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  3. 2023Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learnerscited by 152
  4. 2023ImageBind-LLM: Multi-modality Instruction Tuningcited by 142
  5. 2023Vision-by-Language for Training-Free Compositional Image Retrievalcited by 102
  6. 2023Waffling around for Performance: Visual Classification with Random Words and Broad Conceptscited by 100
  7. 2023Leaving Reality to Imagination: Robust Classification via Generated Datasetscited by 91
  8. 2023Not All Features Matter: Enhancing Few-shot CLIP with Adaptive Prior Refinementcited by 84
  9. 2023Diversity is Definitely Needed: Improving Model-Agnostic Zero-shot Classification via Stable Diffusioncited by 81
  10. 2024No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performancecited by 75
  11. 2024On the test-time zero-shot generalization of vision-language models: Do we really need prompt learning?cited by 65
  12. 2023Diffusion Models for Open-Vocabulary Segmentationcited by 63
38 more
  1. 2024A Hard-to-Beat Baseline for Training-free CLIP-based Adaptationcited by 51
  2. 2024Dual Memory Networks: A Versatile Adaptation Approach for Vision-Language Modelscited by 51
  3. 2023Enhancing CLIP with GPT-4: Harnessing Visual Descriptions as Promptscited by 48
  4. 2023Norm-guided latent space exploration for text-to-image generationcited by 43
  5. 2025Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversioncited by 42
  6. 2023Retrieval-Enhanced Contrastive Vision-Text Modelscited by 36
  7. 2024Generalized Out-of-Distribution Detection and Beyond in Vision Language Model Era: A Surveycited by 36
  8. 2024Quality-Aware Image-Text Alignment for Opinion-Unaware Image Quality Assessmentcited by 31
  9. 2024Just Shift It: Test-Time Prototype Shifting for Zero-Shot Generalization with Vision-Language Modelscited by 31
  10. 2024MoPE-CLIP: Structured Pruning for Efficient Vision-Language Models with Module-wise Pruning Error Metriccited by 30
  11. 2024Few-shot Adaptation of Multi-modal Foundation Models: A Surveycited by 28
  12. 2024Label Propagation for Zero-shot Classification with Vision-Language Modelscited by 28
  13. 2023Intra-Modal Proxy Learning for Zero-Shot Visual Categorization with CLIPcited by 27
  14. 2024AWT: Transferring Vision-Language Models via Augmentation, Weighting, and Transportationcited by 27
  15. 2024Enhancing Zero-Shot Vision Models by Label-Free Prompt Distribution Learning and Bias Correctingcited by 26
  16. 2024How to Merge Your Multimodal Models Over Time?cited by 23
  17. 2025Advances in Multimodal Adaptation and Generalization: From Traditional Approaches to Foundation Modelscited by 23
  18. 2024AMU-Tuning: Effective Logit Bias for CLIP-based Few-shot Learningcited by 22
  19. 2024A Practitioner's Guide to Continual Multimodal Pretrainingcited by 22
  20. 2025LPOSS: Label Propagation Over Patches and Pixels for Open-vocabulary Semantic Segmentationcited by 21
  21. 2024kNN-CLIP: Retrieval Enables Training-Free Segmentation on Continually Expanding Large Vocabulariescited by 20
  22. 2024Active Data Curation Effectively Distills Large-Scale Multimodal Modelscited by 19
  23. 2024Leveraging Cross-Modal Neighbor Representation for Improved CLIP Classificationcited by 18
  24. 2023PerceptionCLIP: Visual Classification by Inferring and Conditioning on Contextscited by 16
  25. 2023Domain Aligned CLIP for Few-shot Classificationcited by 16
  26. 2025Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgettingcited by 16
  27. 20243D Part Segmentation via Geometric Aggregation of 2D Visual Featurescited by 15
  28. 2025ProKeR: A Kernel Perspective on Few-Shot Adaptation of Large Vision-Language Modelscited by 15
  29. 2023Knowledge Transfer from Vision Foundation Models for Efficient Training of Small Task-specific Modelscited by 14
  30. 2024A3lign-DFER: Pioneering Comprehensive Dynamic Affective Alignment for Dynamic Facial Expression Recognition with CLIPcited by 14
  31. 2024Local-Prompt: Extensible Local Prompts for Few-Shot Out-of-Distribution Detectioncited by 14
  32. 2023Follow-Up Differential Descriptions: Language Models Resolve Ambiguities for Image Classificationcited by 13
  33. 2024Learning to Prompt Segment Anything Modelscited by 13
  34. 2023Neural Priming for Sample-Efficient Adaptationcited by 12
  35. 2023Proto-CLIP: Vision-Language Prototypical Network for Few-Shot Learningcited by 12
  36. 2025ProAPO: Progressively Automatic Prompt Optimization for Visual Classificationcited by 12
  37. 2025Logits DeConfusion with CLIP for Few-Shot Learningcited by 11
  38. 2024Understanding Retrieval-Augmented Task Adaptation for Vision-Language Modelscited by 10

50 most cited of 118 All 118 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Robust Fine-Tuning of Vision-Language Models for Domain Generalization
  2. 2023Contrastive Alignment of Vision to Language Through Parameter-Efficient Transfer Learning
  3. 2025Adapting Vision-Language Models Without Labels: A Comprehensive Survey
  4. 2024CLIP-PING: Boosting Lightweight Vision-Language Models with Proximus Intrinsic Neighbors Guidance
  5. 2023Filtering, Distillation, and Hard Negatives for Vision-Language Pre-Training
  6. 2023UniFine: A Unified and Fine-grained Approach for Zero-shot Vision-Language Understanding
  7. 2025Vocabulary-free few-shot learning for Vision-Language Models
  8. 2023Meta-Adapter: An Online Few-shot Learner for Vision-Language Model