mlsanityData to 7 Oct 2026

Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling

arXiv 2111.03930 ↗First posted 6 Nov 2021cs.CV cs.CLcited by 429

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Contrastive Vision-Language Pre-training, known as CLIP, has provided a new paradigm for learning visual representations by using large-scale contrastive image-text pairs. It shows impressive performance on zero-shot knowledge transfer to downstream tasks. To further enhance CLIP's few-shot capability, CLIP-Adapter proposed to fine-tune a lightweight residual feature adapter and significantly improves the performance for few-shot classification. However, such a process still needs extra training and computational resources. In this paper, we propose Training-Free CLIP-Adapter (Tip-Adapter), which not only inherits CLIP's training-free advantage but also performs comparably or even better than CLIP-Adapter. Tip-Adapter does not require any back propagation for training the adapter, but creates the weights by a key-value cache model constructed from the few-shot training set. In this non-parametric manner, Tip-Adapter acquires well-performed adapter weights without any training, which is both efficient and effective. Moreover, the performance of Tip-Adapter can be further boosted by fine-tuning such properly initialized adapter for only a few epochs with super-fast convergence speed. We conduct extensive experiments of few-shot classification on ImageNet and other 10 datasets to demonstrate the superiority of proposed Tip-Adapter. The code will be released at urlhttps://github.com/gaopengcuhk/Tip-Adapter.

Key facts. Cited by 429 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 53 of its 70 references that are in the map. First posted on arXiv 6 Nov 2021.

Builds on

53 of 70 references.

Closest ancestors · 2

Same direction, newest first

  1. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
  2. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 51

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  6. 2020Language Models are Few-Shot Learnerscited by 32,199
  7. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  8. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  10. 2020End-to-End Object Detection with Transformerscited by 7,384
  11. 2017Mask R-CNNcited by 7,259
  12. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
38 more
  1. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  2. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  3. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  5. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  6. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  7. 2016Pointer Sentinel Mixture Modelscited by 2,992
  8. 2015VQA: Visual Question Answeringcited by 2,489
  9. 2017Billion-scale similarity search with GPUscited by 2,477
  10. 2016Matching Networks for One Shot Learningcited by 2,246
  11. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  12. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  13. 2013Describing Textures in the Wildcited by 2,110
  14. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  15. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  16. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  17. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  18. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  19. 2017Learning to Compare: Relation Network for Few-Shot Learningcited by 1,139
  20. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  21. 2021Robust fine-tuning of zero-shot modelscited by 785
  22. 2017Focal Loss for Dense Object Detectioncited by 749
  23. 2019A Closer Look at Few-shot Classificationcited by 679
  24. 2020Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?cited by 435
  25. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 427
  26. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  27. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  28. 2019A Baseline for Few-Shot Image Classificationcited by 298
  29. 2019Attention on Attention for Image Captioningcited by 289
  30. 2018Bilinear Attention Networkscited by 277
  31. 2016Image Captioning with Semantic Attentioncited by 260
  32. 2018Scaling Neural Machine Translationcited by 204
  33. 2021Fast Convergence of DETR with Spatially Modulated Co-Attentioncited by 176
  34. 2018Dynamic Fusion with Intra- and Inter- Modality Attention Flow for Visual Question Answeringcited by 113
  35. 2021Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learningcited by 112
  36. 2019Augmenting Self-attention with Persistent Memorycited by 92
  37. 2017Unbounded cache model for online language modeling with open vocabularycited by 37
  38. 2021Dual-stream Network for Visual Recognitioncited by 33

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 53; 2023: 107; 2024: 116; 2025: 107; 2026: 40.

429 papers.

Sort:

By area:

  1. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  2. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  3. 2021Grounded Language-Image Pre-trainingcited by 1,147
  4. 2022Exploring CLIP for Assessing the Look and Feel of Imagescited by 974
  5. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  6. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  7. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  8. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  9. 2022Vision Transformer Adapter for Dense Predictionscited by 564
  10. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 530
  11. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 461
  12. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Modelscited by 379
38 more
  1. 2023Repurposing Diffusion-Based Image Generators for Monocular Depth Estimationcited by 323
  2. 2022Expanding Language-Image Pretrained Models for General Video Recognitioncited by 320
  3. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 319
  4. 2021Prompting Visual-Language Models for Efficient Video Understandingcited by 309
  5. 2023Side Adapter Network for Open-Vocabulary Semantic Segmentationcited by 296
  6. 2023Self-regulating Prompts: Foundational Model Adaptation without Forgettingcited by 272
  7. 2023Personalize Segment Anything Model with One Shotcited by 250
  8. 2022LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learningcited by 228
  9. 2022Delving into Out-of-Distribution Detection with Vision-Language Representationscited by 215
  10. 2023CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentationcited by 215
  11. 2024Motion-I2V: Consistent and Controllable Image-to-Video Generation with Explicit Motion Modelingcited by 209
  12. 2024Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapterscited by 199
  13. 2022Patching open-vocabulary models by interpolating weightscited by 187
  14. 2022PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learningcited by 186
  15. 2022Frozen CLIP Models are Efficient Video Learnerscited by 181
  16. 2022Fine-tuned CLIP Models are Efficient Video Learnerscited by 178
  17. 2023Towards Open Vocabulary Learning: A Surveycited by 165
  18. 2022CLIP2Point: Transfer CLIP to Point Cloud Classification with Image-Depth Pre-trainingcited by 159
  19. 2023Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learnerscited by 152
  20. 2023RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote Sensingcited by 150
  21. 2023Diverse Data Augmentation with Diffusions for Effective Test-time Prompt Tuningcited by 150
  22. 2022Finetune like you pretrain: Improved finetuning of zero-shot vision modelscited by 149
  23. 2023DriveAdapter: Breaking the Coupling Barrier of Perception and Planning in End-to-End Autonomous Drivingcited by 142
  24. 2023ImageBind-LLM: Multi-modality Instruction Tuningcited by 142
  25. 2022Unsupervised Prompt Learning for Vision-Language Modelscited by 125
  26. 2021Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Taskscited by 124
  27. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
  28. 2022Backbone is All Your Need: A Simplified Architecture for Visual Object Trackingcited by 119
  29. 2022SuS-X: Training-Free Name-Only Transfer of Vision-Language Modelscited by 118
  30. 2023LoCoOp: Few-Shot Out-of-Distribution Detection via Prompt Learningcited by 107
  31. 2022CALIP: Zero-Shot Enhancement of CLIP with Parameter-free Attentioncited by 103
  32. 2022DualCoOp: Fast Adaptation to Multi-Label Recognition with Limited Annotationscited by 102
  33. 2022Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIPcited by 97
  34. 2023CLIPood: Generalizing CLIP to Out-of-Distributionscited by 96
  35. 2022"This is my unicorn, Fluffy": Personalizing frozen vision-language representationscited by 95
  36. 2025Lumina-Image 2.0: A Unified and Efficient Image Generative Frameworkcited by 93
  37. 2023Fine-tuning can cripple your foundation model; preserving features may be the solutioncited by 92
  38. 2023Multimodality Helps Unimodality: Cross-Modal Few-Shot Learning with Multimodal Modelscited by 91

50 most cited of 429 All 429 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification
  2. 2024CLIP's Visual Embedding Projector is a Few-shot Cornucopia
  3. 2023Contrastive Alignment of Vision to Language Through Parameter-Efficient Transfer Learning
  4. 2024Fully Fine-tuned CLIP Models are Efficient Few-Shot Learners
  5. 2024CLIP-PING: Boosting Lightweight Vision-Language Models with Proximus Intrinsic Neighbors Guidance
  6. 2023CLIP meets Model Zoo Experts: Pseudo-Supervision for Visual Enhancement
  7. 2025CLIPTTA: Robust Contrastive Vision-Language Test-Time Adaptation
  8. 2024Rethinking Visual Content Refinement in Low-Shot CLIP Adaptation