mlsanityData to 7 Oct 2026

What does a platypus look like? Generating customized prompts for zero-shot image classification

arXiv 2209.03320 ↗First posted 7 Sep 2022cs.CV cs.LGcited by 263

Not on the map. Why?

Cited by 129 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Open-vocabulary models are a promising new paradigm for image classification. Unlike traditional classification models, open-vocabulary models classify among any arbitrary set of categories specified with natural language during inference. This natural language, called "prompts", typically consists of a set of hand-written templates (e.g., "a photo of a {}") which are completed with each of the category names. This work introduces a simple method to generate higher accuracy prompts, without relying on any explicit knowledge of the task domain and with far fewer hand-constructed sentences. To achieve this, we combine open-vocabulary models with large language models (LLMs) to create Customized Prompts via Language models (CuPL, pronounced "couple"). In particular, we leverage the knowledge contained in LLMs in order to generate many descriptive sentences that contain important discriminating characteristics of the image categories. This allows the model to place a greater importance on these regions in the image when making predictions. We find that this straightforward and general approach improves accuracy on a range of zero-shot image classification benchmarks, including over one percentage point gain on ImageNet. Finally, this simple baseline requires no additional training and remains completely zero-shot. Code available at https://github.com/sarahpratt/CuPL.

Key facts. Cited by 263 papers in the map (data through 7 Oct 2026). Builds on 47 of its 74 references that are in the map. First posted on arXiv 7 Sep 2022.

Builds on

47 of 74 references.

References · 47

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  10. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  11. 2013Describing Textures in the Wildcited by 2,110
  12. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
35 more
  1. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  2. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  3. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  4. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  5. 2019How Can We Know What Language Models Know?cited by 953
  6. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  7. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  8. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 791
  9. 2021Robust fine-tuning of zero-shot modelscited by 785
  10. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  11. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 530
  12. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 376
  13. 2020Are we done with ImageNet?cited by 329
  14. 2019A Short Note on the Kinetics-700 Human Action Datasetcited by 308
  15. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 298
  16. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 288
  17. 2020AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Promptscited by 246
  18. 2021Generated Knowledge Prompting for Commonsense Reasoningcited by 213
  19. 2021Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classificationcited by 183
  20. 2016Learning Deep Representations of Fine-grained Visual Descriptionscited by 182
  21. 2022Unsupervised Prompt Learning for Vision-Language Modelscited by 125
  22. 2021Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learnerscited by 115
  23. 2022Algorithms to estimate Shapley value feature attributionscited by 114
  24. 2021PartImageNet: A Large, High-Quality Dataset of Partscited by 107
  25. 2021Sampling Permutations for Shapley Value Estimationcited by 101
  26. 2022Language Models Can See: Plugging Visual Controls in Text Generationcited by 89
  27. 2022K-LITE: Learning Transferable Visual Models with External Knowledgecited by 79
  28. 2017Fine-graind Image Classification via Combining Vision and Languagecited by 51
  29. 2017Link the head to the "beak": Zero Shot Learning from Noisy Text Description at Part Precisioncited by 46
  30. 2022Learning to Estimate Shapley Values with Vision Transformerscited by 44
  31. 2022Is a Caption Worth a Thousand Images? A Controlled Study for Representation Learningcited by 39
  32. 2022Multimodal Knowledge Alignment with Reinforcement Learningcited by 30
  33. 2020ZEST: Zero-shot Learning from Text Descriptions using Textual Similarity and Visual Summarizationcited by 22
  34. 2021Large-Scale Zero-Shot Image Classification from Rich and Diverse Textual Descriptionscited by 21
  35. 2020Attributes-Guided and Pure-Visual Attention Alignment for Few-Shot Recognitioncited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 56; 2024: 87; 2025: 78; 2026: 36.

263 papers.

Sort:

By area:

  1. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  2. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  3. 2022Visual Classification via Description from Large Language Modelscited by 331
  4. 2022Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classificationcited by 310
  5. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Modelscited by 297
  6. 2022PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learningcited by 186
  7. 2023Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learnerscited by 152
  8. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 134
  9. 2022SuS-X: Training-Free Name-Only Transfer of Vision-Language Modelscited by 118
  10. 2025EgoVLA: Learning Vision-Language-Action Models from Egocentric Human Videoscited by 116
  11. 2023Vision-by-Language for Training-Free Compositional Image Retrievalcited by 102
  12. 2023Waffling around for Performance: Visual Classification with Random Words and Broad Conceptscited by 100
38 more
  1. 20233D-GPT: Procedural 3D Modeling with Large Language Modelscited by 91
  2. 2023Not All Features Matter: Enhancing Few-shot CLIP with Adaptive Prior Refinementcited by 84
  3. 2024Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunitiescited by 83
  4. 2023Diversity is Definitely Needed: Improving Model-Agnostic Zero-shot Classification via Stable Diffusioncited by 81
  5. 2023CHiLS: Zero-Shot Image Classification with Hierarchical Label Setscited by 79
  6. 2023Learning Concise and Descriptive Attributes for Visual Recognitioncited by 73
  7. 2023Open-Set Image Tagging with Multi-Grained Text Supervisioncited by 67
  8. 2024Zero-Shot ECG Classification with Multimodal Learning and Test-time Clinical Knowledge Enhancementcited by 66
  9. 2024On the test-time zero-shot generalization of vision-language models: Do we really need prompt learning?cited by 65
  10. 2024The Neglected Tails in Vision-Language Modelscited by 63
  11. 2023Multi-Modal Classifiers for Open-Vocabulary Object Detectioncited by 53
  12. 2024Improved Zero-Shot Classification by Adapting VLMs with Text Descriptionscited by 52
  13. 2024Dual Memory Networks: A Versatile Adaptation Approach for Vision-Language Modelscited by 51
  14. 2023Enhancing CLIP with GPT-4: Harnessing Visual Descriptions as Promptscited by 48
  15. 2023LaFTer: Label-Free Tuning of Zero-shot Classifier using Language and Unlabeled Image Collectionscited by 43
  16. 2023CLOVA: A Closed-Loop Visual Assistant with Tool Usage and Updatecited by 42
  17. 2023SemiVL: Semi-Supervised Semantic Segmentation with Vision-Language Guidancecited by 41
  18. 2023ArGue: Attribute-Guided Prompt Tuning for Vision-Language Modelscited by 39
  19. 2024Learning to Prompt with Text Only Supervision for Vision-Language Modelscited by 37
  20. 2023CoTDet: Affordance Knowledge Prompting for Task Driven Object Detectioncited by 36
  21. 2023Language Models as Black-Box Optimizers for Vision-Language Modelscited by 36
  22. 2024Dual Prototype Evolving for Test-Time Generalization of Vision-Language Modelscited by 36
  23. 2023Exposing and Mitigating Spurious Correlations for Cross-Modal Retrievalcited by 35
  24. 2023Towards Calibrated Robust Fine-Tuning of Vision-Language Modelscited by 35
  25. 2023Distilling Large Vision-Language Model with Out-of-Distribution Generalizabilitycited by 33
  26. 2023Collaborating Foundation Models for Domain Generalized Semantic Segmentationcited by 33
  27. 20253D Aware Region Prompted Vision Language Modelcited by 31
  28. 2024Visual-Text Cross Alignment: Refining the Similarity Score in Vision-Language Modelscited by 30
  29. 2023AttrSeg: Open-Vocabulary Semantic Segmentation via Attribute Decomposition-Aggregationcited by 29
  30. 2022Action-GPT: Leveraging Large-scale Language Models for Improved and Generalized Action Generationcited by 28
  31. 2024Few-shot Adaptation of Multi-modal Foundation Models: A Surveycited by 28
  32. 2024Meta-Prompting for Automating Zero-shot Visual Recognition with LLMscited by 28
  33. 2024Label Propagation for Zero-shot Classification with Vision-Language Modelscited by 28
  34. 2024AWT: Transferring Vision-Language Models via Augmentation, Weighting, and Transportationcited by 27
  35. 2024Enhancing Zero-Shot Vision Models by Label-Free Prompt Distribution Learning and Bias Correctingcited by 26
  36. 2023Active Prompt Learning in Vision Language Modelscited by 25
  37. 2024Democratizing Fine-grained Visual Recognition with Large Language Modelscited by 25
  38. 2024CLAP4CLIP: Continual Learning with Probabilistic Finetuning for Vision-Language Modelscited by 25

50 most cited of 263 All 263 (JSON)