mlsanityData to 7 Oct 2026

Prompt Distribution Learning

arXiv 2205.03340 ↗First posted 6 May 2022cs.CVcited by 235

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

We present prompt distribution learning for effectively adapting a pre-trained vision-language model to address downstream recognition tasks. Our method not only learns low-bias prompts from a few samples but also captures the distribution of diverse prompts to handle the varying visual representations. In this way, we provide high-quality task-related content for facilitating recognition. This prompt distribution learning is realized by an efficient approach that learns the output embeddings of prompts instead of the input embeddings. Thus, we can employ a Gaussian distribution to model them effectively and derive a surrogate loss for efficient training. Extensive experiments on 12 datasets demonstrate that our method consistently and significantly outperforms existing methods. For example, with 1 sample per category, it relatively improves the average result by 9.1% compared to human-crafted prompts.

Key facts. Cited by 235 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 29 of its 51 references that are in the map. First posted on arXiv 6 May 2022.

Builds on

29 of 51 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 28

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  10. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  11. 2013Describing Textures in the Wildcited by 2,110
  12. 2019Language Models as Knowledge Bases?cited by 1,922
16 more
  1. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  2. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  3. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  4. 2019How Can We Know What Language Models Know?cited by 953
  5. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  6. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  7. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  8. 2020Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?cited by 435
  9. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  10. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
  11. 2021PTR: Prompt Tuning with Rules for Text Classificationcited by 238
  12. 2015Learning Visual Features from Large Weakly Supervised Datacited by 165
  13. 2020Learning Visual Representations with Caption Annotationscited by 123
  14. 2016Learning Visual N-Grams from Web Datacited by 95
  15. 2020Regularizing Deep Networks with Semantic Data Augmentationcited by 73
  16. 2014The Application of Two-level Attention Models in Deep Convolutional Neural Network for Fine-grained Image Classificationcited by 67

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 23; 2023: 68; 2024: 69; 2025: 50; 2026: 25.

235 papers.

Sort:

By area:

  1. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  2. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  3. 2022Prompt-aligned Gradient for Prompt Tuningcited by 339
  4. 2023Visual-Language Prompt Tuning with Knowledge-guided Context Optimizationcited by 307
  5. 2023Self-regulating Prompts: Foundational Model Adaptation without Forgettingcited by 272
  6. 2022Unified Vision and Language Prompt Learningcited by 145
  7. 2022Neural Prompt Searchcited by 133
  8. 2023DDCoT: Duty-Distinct Chain-of-Thought Prompting for Multimodal Reasoning in Language Modelscited by 124
  9. 2022Task Residual for Tuning Vision-Language Modelscited by 122
  10. 2022SuS-X: Training-Free Name-Only Transfer of Vision-Language Modelscited by 118
  11. 2024Low-Rank Few-Shot Adaptation of Vision-Language Modelscited by 103
  12. 2022DualCoOp: Fast Adaptation to Multi-Label Recognition with Limited Annotationscited by 102
38 more
  1. 2023Cheap and Quick: Efficient Vision-Language Instruction Tuning for Large Language Modelscited by 100
  2. 2023Waffling around for Performance: Visual Classification with Random Words and Broad Conceptscited by 100
  3. 2023CLIPood: Generalizing CLIP to Out-of-Distributionscited by 96
  4. 2023Multimodality Helps Unimodality: Cross-Modal Few-Shot Learning with Multimodal Modelscited by 91
  5. 2023Learning without Forgetting for Vision-Language Modelscited by 79
  6. 2023Consistency-guided Prompt Learning for Vision-Language Modelscited by 73
  7. 2023GraphAdapter: Tuning Vision-Language Models With Dual Knowledge Graphcited by 69
  8. 2024On the test-time zero-shot generalization of vision-language models: Do we really need prompt learning?cited by 65
  9. 2022PLOT: Prompt Learning with Optimal Transport for Vision-Language Modelscited by 64
  10. 2023BadCLIP: Trigger-Aware Prompt Learning for Backdoor Attacks on CLIPcited by 63
  11. 2024Perceive-IR: Learning to Perceive Degradation Better for All-in-One Image Restorationcited by 60
  12. 2023PromptStyler: Prompt-driven Style Generation for Source-free Domain Generalizationcited by 59
  13. 2023Sentence-level Prompts Benefit Composed Image Retrievalcited by 59
  14. 2022LASP: Text-to-Text Optimization for Language-Aware Soft Prompting of Vision & Language Modelscited by 58
  15. 2023Diversity-Aware Meta Visual Promptingcited by 57
  16. 2022I2MVFormer: Large Language Model Generated Multi-View Document Supervision for Zero-Shot Image Classificationcited by 54
  17. 2022Dual Modality Prompt Tuning for Vision-Language Pre-Trained Modelcited by 51
  18. 2022Bayesian Prompt Learning for Image-Language Model Generalizationcited by 51
  19. 2022Multitask Vision-Language Prompt Tuningcited by 51
  20. 2022Texts as Images in Prompt Tuning for Multi-Label Image Recognitioncited by 51
  21. 2023Improving Zero-Shot Generalization for CLIP with Synthesized Promptscited by 51
  22. 2024A Hard-to-Beat Baseline for Training-free CLIP-based Adaptationcited by 51
  23. 2024Dual Memory Networks: A Versatile Adaptation Approach for Vision-Language Modelscited by 51
  24. 2024Global and Local Prompts Cooperation via Optimal Transport for Federated Learningcited by 45
  25. 2023Revisiting the Parameter Efficiency of Adapters from the Perspective of Precision Redundancycited by 44
  26. 2023LaFTer: Label-Free Tuning of Zero-shot Classifier using Language and Unlabeled Image Collectionscited by 43
  27. 2023Prompt-In-Prompt Learning for Universal Image Restorationcited by 43
  28. 2023ID-like Prompt Learning for Few-Shot Out-of-Distribution Detectioncited by 42
  29. 2023CLOVA: A Closed-Loop Visual Assistant with Tool Usage and Updatecited by 42
  30. 2022Understanding and Mitigating Overfitting in Prompt Tuning for Vision-Language Modelscited by 40
  31. 2023Black Box Few-Shot Adaptation for Vision-Language modelscited by 39
  32. 2023ArGue: Attribute-Guided Prompt Tuning for Vision-Language Modelscited by 39
  33. 2024Few-Shot Adversarial Prompt Learning on Vision-Language Modelscited by 39
  34. 2024Learning to Prompt with Text Only Supervision for Vision-Language Modelscited by 37
  35. 2023StyLIP: Multi-Scale Style-Conditioned Prompt Learning for CLIP-based Domain Generalizationcited by 36
  36. 2023Language Models as Black-Box Optimizers for Vision-Language Modelscited by 36
  37. 2024Boosting Vision-Language Models with Transductioncited by 36
  38. 2024Generalized Out-of-Distribution Detection and Beyond in Vision Language Model Era: A Surveycited by 36

50 most cited of 235 All 235 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Revisiting Prompt Pretraining of Vision-Language Models
  2. 2024Aggregate-and-Adapt Natural Language Prompts for Downstream Generalization of CLIP
  3. 2023Towards Robust Prompts on Vision-Language Models
  4. 2024Make Prompts Adaptable: Bayesian Modeling for Vision-Language Prompt Learning with Data-Dependent Prior
  5. 2023PRE: Vision-Language Prompt Learning with Reparameterization Encoder
  6. 2026What Does Prompt Learning Change? -A Natural-Language Concept Analysis of Vision-Language Models
  7. 2023LaViP:Language-Grounded Visual Prompts
  8. 2024PLPP: Prompt Learning with Perplexity Is Self-Distillation for Vision-Language Models