mlsanityData to 7 Oct 2026

CLIP-Adapter: Better Vision-Language Models with Feature Adapters

arXiv 2110.04544 ↗First posted 9 Oct 2021cs.CV cs.CLcited by 1,021

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Large-scale contrastive vision-language pre-training has shown significant progress in visual representation learning. Unlike traditional visual systems trained by a fixed set of discrete labels, a new paradigm was introduced in citeradford2021learning to directly learn to align images with raw texts in an open-vocabulary setting. On downstream tasks, a carefully chosen text prompt is employed to make zero-shot predictions.~To avoid non-trivial prompt engineering, context optimization citezhou2021coop has been proposed to learn continuous vectors as task-specific prompts with few-shot training examples.~In this paper, we show that there is an alternative path to achieve better vision-language models other than prompt tuning.~While prompt tuning is for the textual inputs, we propose CLIP-Adapter to conduct fine-tuning with feature adapters on either visual or language branch. Specifically, CLIP-Adapter adopts an additional bottleneck layer to learn new features and performs residual-style feature blending with the original pre-trained features.~As a consequence, CLIP-Adapter is able to outperform context optimization while maintains a simple design. Experiments and extensive ablation studies on various visual classification tasks demonstrate the effectiveness of our approach. Code is released at t https://github.com/gaopengcuhk/CLIP-Adapter.

Key facts. Cited by 1,021 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 49 of its 80 references that are in the map. First posted on arXiv 9 Oct 2021.

Builds on

49 of 80 references.

Closest ancestors · 2

Same direction, newest first

  1. 2021CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models24 Sep
  2. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 47

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  7. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  8. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  9. 2020End-to-End Object Detection with Transformerscited by 7,384
  10. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  11. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  12. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
35 more
  1. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  2. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  4. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  5. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  6. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  7. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  8. 2013Describing Textures in the Wildcited by 2,110
  9. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  10. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  11. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  12. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  13. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  14. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  15. 2019Natural Adversarial Examplescited by 1,401
  16. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  17. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  18. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  19. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  20. 2019How Can We Know What Language Models Know?cited by 953
  21. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  22. 2021Robust fine-tuning of zero-shot modelscited by 785
  23. 2021GPT Understands, Toocited by 715
  24. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  25. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  26. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  27. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 338
  28. 2018Bilinear Attention Networkscited by 277
  29. 2020AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Promptscited by 246
  30. 2021PPT: Pre-trained Prompt Tuning for Few-shot Learningcited by 232
  31. 2019UNITER: UNiversal Image-TExt Representation Learningcited by 225
  32. 2021Fast Convergence of DETR with Spatially Modulated Co-Attentioncited by 176
  33. 2018Dynamic Fusion with Intra- and Inter- Modality Attention Flow for Visual Question Answeringcited by 113
  34. 2021Container: Context Aggregation Networkcited by 37
  35. 2021Dual-stream Network for Visual Recognitioncited by 33

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 10; 2022: 75; 2023: 209; 2024: 295; 2025: 284; 2026: 148.

1,021 papers.

Sort:

By area:

  1. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  2. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  3. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  4. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  5. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  6. 2021Grounded Language-Image Pre-trainingcited by 1,147
  7. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,099
  8. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  9. 2021Generalized Out-of-Distribution Detection: A Surveycited by 796
  10. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  11. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  12. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
38 more
  1. 2022Vision Transformer Adapter for Dense Predictionscited by 564
  2. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 530
  3. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  4. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 461
  5. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 429
  6. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classificationcited by 355
  7. 2022Prompt-aligned Gradient for Prompt Tuningcited by 339
  8. 2023Repurposing Diffusion-Based Image Generators for Monocular Depth Estimationcited by 323
  9. 2022Expanding Language-Image Pretrained Models for General Video Recognitioncited by 320
  10. 2022Is synthetic data from generative models ready for image recognition?cited by 320
  11. 2022Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classificationcited by 310
  12. 2021Prompting Visual-Language Models for Efficient Video Understandingcited by 309
  13. 2023Visual-Language Prompt Tuning with Knowledge-guided Context Optimizationcited by 307
  14. 2022GLIPv2: Unifying Localization and Vision-Language Understandingcited by 275
  15. 2023Self-regulating Prompts: Foundational Model Adaptation without Forgettingcited by 272
  16. 2023Unleashing Text-to-Image Diffusion Models for Visual Perceptioncited by 266
  17. 2022LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learningcited by 228
  18. 2023CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentationcited by 215
  19. 2024Motion-I2V: Consistent and Controllable Image-to-Video Generation with Explicit Motion Modelingcited by 209
  20. 2024Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapterscited by 199
  21. 2022Patching open-vocabulary models by interpolating weightscited by 187
  22. 2022PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learningcited by 186
  23. 2022Frozen CLIP Models are Efficient Video Learnerscited by 181
  24. 2022CLIP-ReID: Exploiting Vision-Language Model for Image Re-Identification without Concrete Text Labelscited by 177
  25. 2023SCLIP: Rethinking Self-Attention for Dense Vision-Language Inferencecited by 169
  26. 2024Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streamscited by 160
  27. 2023Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learnerscited by 152
  28. 2023RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote Sensingcited by 150
  29. 2023Diverse Data Augmentation with Diffusions for Effective Test-time Prompt Tuningcited by 150
  30. 2022Finetune like you pretrain: Improved finetuning of zero-shot vision modelscited by 149
  31. 2023Debiasing Vision-Language Models via Biased Promptscited by 147
  32. 2023DriveAdapter: Breaking the Coupling Barrier of Perception and Planning in End-to-End Autonomous Drivingcited by 142
  33. 2022ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Modelscited by 141
  34. 2024Efficient Test-Time Adaptation of Vision-Language Modelscited by 138
  35. 2022Neural Prompt Searchcited by 133
  36. 2023CORA: Adapting CLIP for Open-Vocabulary Detection with Region Prompting and Anchor Pre-Matchingcited by 133
  37. 2023CLIPN for Zero-Shot OOD Detection: Teaching CLIP to Say Nocited by 130
  38. 2022Unsupervised Prompt Learning for Vision-Language Modelscited by 125

50 most cited of 1,021 All 1,021 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Fully Fine-tuned CLIP Models are Efficient Few-Shot Learners
  2. 2023Contrastive Alignment of Vision to Language Through Parameter-Efficient Transfer Learning
  3. 2024CLIPArTT: Adaptation of CLIP to New Domains at Test Time
  4. 2023Class Incremental Learning with Pre-trained Vision-Language Models
  5. 2025Post-pre-training for Modality Alignment in Vision-Language Foundation Models
  6. 2023Improved baselines for vision-language pre-training
  7. 2024Rethinking Visual Content Refinement in Low-Shot CLIP Adaptation
  8. 2022CLIP-TD: CLIP Targeted Distillation for Vision-Language Tasks