mlsanityData to 7 Oct 2026

A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Models

arXiv 2110.08484 ↗First posted 16 Oct 2021cs.CV cs.CLcited by 101

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Large pre-trained vision-language (VL) models can learn a new task with a handful of examples and generalize to a new task without fine-tuning. However, these VL models are hard to deploy for real-world applications due to their impractically huge sizes and slow inference speed. To solve this limitation, we study prompt-based low-resource learning of VL tasks with our proposed method, FewVLM, relatively smaller than recent few-shot learners. For FewVLM, we pre-train a sequence-to-sequence transformer model with prefix language modeling (PrefixLM) and masked language modeling (MaskedLM). Furthermore, we analyze the effect of diverse prompts for few-shot tasks. Experimental results on VQA show that FewVLM with prompt-based learning outperforms Frozen which is 31x larger than FewVLM by 18.2% point and achieves comparable results to a 246x larger model, PICa. In our analysis, we observe that (1) prompts significantly affect zero-shot performance but marginally affect few-shot performance, (2) models with noisy prompts learn as quickly as hand-crafted prompts given larger training data, and (3) MaskedLM helps VQA tasks while PrefixLM boosts captioning performance. Our code is publicly available at urlhttps://github.com/woojeongjin/FewVLM

Key facts. Cited by 101 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 35 of its 43 references that are in the map. First posted on arXiv 16 Oct 2021.

Builds on

35 of 43 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  9. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  10. 2016Matching Networks for One Shot Learningcited by 2,246
  11. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  12. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
23 more
  1. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  2. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  3. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  4. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  5. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  6. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  7. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  8. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  9. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  10. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  11. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  12. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  13. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  14. 2018nocaps: novel object captioning at scalecited by 444
  15. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  16. 2021True Few-Shot Learning with Language Modelscited by 343
  17. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  18. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 298
  19. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  20. 2019UNITER: UNiversal Image-TExt Representation Learningcited by 225
  21. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learningcited by 187
  22. 2020Adversarial Feature Hallucination Networks for Few-Shot Learningcited by 110
  23. 2020Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Modelscited by 90

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 21; 2023: 40; 2024: 17; 2025: 22; 2026: 1.

101 papers.

Sort:

By area:

  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  2. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  3. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  4. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  5. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  6. 2022Prompt-aligned Gradient for Prompt Tuningcited by 339
  7. 2022TabLLM: Few-shot Classification of Tabular Data with Large Language Modelscited by 273
  8. 2022Making Large Language Models Better Reasoners with Step-Aware Verifiercited by 231
  9. 2022From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Modelscited by 114
  10. 2023Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Modelscited by 112
  11. 2022Language Models are General-Purpose Interfacescited by 96
  12. 2022Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Trainingcited by 88
38 more
  1. 2022SmallCap: Lightweight Image Captioning Prompted with Retrieval Augmentationcited by 82
  2. 2022Learning to Compose Soft Prompts for Compositional Zero-Shot Learningcited by 66
  3. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directionscited by 66
  4. 2023Towards Language Models That Can See: Computer Vision Through the LENS of Natural Languagecited by 53
  5. 2023See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoningcited by 49
  6. 2023Modular Visual Question Answering via Code Generationcited by 47
  7. 2022MAPL: Parameter-Efficient Adaptation of Unimodal Pre-Trained Models for Vision-Language Few-Shot Promptingcited by 41
  8. 2024Learning to Prompt with Text Only Supervision for Vision-Language Modelscited by 37
  9. 2023eP-ALM: Efficient Perceptual Augmentation of Language Modelscited by 33
  10. 2023Can Multimodal Large Language Models Truly Perform Multimodal In-Context Learning?cited by 31
  11. 2023Prismer: A Vision-Language Model with Multi-Task Expertscited by 28
  12. 2023Bootstrapping Vision-Language Learning with Decoupled Language Pre-trainingcited by 28
  13. 2023From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunitiescited by 24
  14. 2022Prompt-based Learning for Unpaired Image Captioningcited by 23
  15. 2023Compositional Prompt Tuning with Motion Cues for Open-vocabulary Video Relation Detectioncited by 22
  16. 2023Meta Learning to Bridge Vision and Language Models for Multimodal Few-Shot Learningcited by 20
  17. 2024Benchmarking In-the-wild Multimodal Disease Recognition and A Versatile Baselinecited by 19
  18. 2023LMCap: Few-shot Multilingual Image Captioning by Retrieval Augmented Language Model Promptingcited by 17
  19. 2025Time Blindness: Why Video-Language Models Can't See What Humans Can?cited by 17
  20. 2023GIPCOL: Graph-Injected Soft Prompting for Compositional Zero-Shot Learningcited by 16
  21. 2022PSP: Pre-trained Soft Prompts for Few-Shot Abstractive Summarizationcited by 14
  22. 2023MAGVLT: Masked Generative Vision-and-Language Transformercited by 14
  23. 2023Investigating Prompting Techniques for Zero- and Few-Shot Visual Question Answeringcited by 14
  24. 2024Towards Truly Zero-shot Compositional Visual Reasoning with LLMs as Programmerscited by 14
  25. 2024X-Former: Unifying Contrastive and Reconstruction Learning for MLLMscited by 14
  26. 2024Leveraging Large Vision-Language Model as User Intent-aware Encoder for Composed Image Retrievalcited by 13
  27. 2022Prompting through Prototype: A Prototype-based Prompt Learning on Pretrained Vision-Language Modelscited by 12
  28. 2023Dynamic Prompting: A Unified Framework for Prompt Tuningcited by 12
  29. 2024Enhancing Vision-Language Pre-training with Rich Supervisionscited by 12
  30. 2022What is Where by Looking: Weakly-Supervised Open-World Phrase-Grounding without Text Inputscited by 11
  31. 2023A Multi-Modal Context Reasoning Approach for Conditional Inference on Joint Textual and Visual Cluescited by 11
  32. 2023Zero-shot Visual Question Answering with Language Model Feedbackcited by 11
  33. 2024ComDrive: Comfort-Oriented End-to-End Autonomous Drivingcited by 11
  34. 2023S3C: Semi-Supervised VQA Natural Language Explanation via Self-Critical Learningcited by 10
  35. 2025How to Teach Large Multimodal Models New Skillscited by 10
  36. 2022Declaration-based Prompt Tuning for Visual Question Answeringcited by 9
  37. 2024Toward Interactive Regional Understanding in Vision-Large Language Modelscited by 9
  38. 2024Paraphrase and Aggregate with Large Language Models for Minimizing Intent Classification Errorscited by 9

50 most cited of 101 All 101 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Low-Rank Few-Shot Adaptation of Vision-Language Models
  2. 2024Revisiting Prompt Pretraining of Vision-Language Models
  3. 2023Language Models as Black-Box Optimizers for Vision-Language Models
  4. 2025Modular Prompt Learning Improves Vision-Language Models
  5. 2025Vocabulary-free few-shot learning for Vision-Language Models
  6. 2024Tuning Vision-Language Models with Candidate Labels by Prompt Alignment
  7. 2025Accelerating Conditional Prompt Learning via Masked Image Modeling for Vision-Language Models
  8. 2025Weighted Multi-Prompt Learning with Description-free Large Language Model Distillation