mlsanityData to 14 Oct 2026

Prompt Pre-Training with Twenty-Thousand Classes for Open-Vocabulary Visual Recognition

arXiv 2304.04704 ↗First posted 10 Apr 2023cs.CV cs.AI cs.CLcited by 35

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

This work proposes POMP, a prompt pre-training method for vision-language models. Being memory and computation efficient, POMP enables the learned prompt to condense semantic information for a rich set of visual concepts with over twenty-thousand classes. Once pre-trained, the prompt with a strong transferable ability can be directly plugged into a variety of visual recognition tasks including image classification, semantic segmentation, and object detection, to boost recognition performances in a zero-shot manner. Empirical evaluation shows that POMP achieves state-of-the-art performances on 21 datasets, e.g., 67.0% average accuracy on 10 classification datasets (+3.1% compared to CoOp) and 84.4 hIoU on open-vocabulary Pascal VOC segmentation (+6.9 compared to ZSSeg). Our code is available at https://github.com/amazon-science/prompt-pretraining.

Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 50 of its 71 references that are in the map. First posted on arXiv 10 Apr 2023.

Builds on

50 of 71 references.

Closest ancestors · 6

Same direction, newest first

  1. 2022Unified Vision and Language Prompt Learning13 Oct
  2. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models15 Sep
  3. 2022Prompt Distribution Learning6 May
  4. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  5. 2021Decoupling Zero-Shot Semantic Segmentation15 Dec
  6. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 44

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  9. 2013Describing Textures in the Wildcited by 2,125
  10. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  11. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  12. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
32 more
  1. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  2. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  3. 2019Natural Adversarial Examplescited by 1,408
  4. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  5. 2019Contrastive Multiview Codingcited by 1,301
  6. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,217
  7. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,152
  8. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  9. 2018CosFace: Large Margin Cosine Loss for Deep Face Recognitioncited by 971
  10. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 864
  11. 2019Big Transfer (BiT): General Visual Representation Learningcited by 769
  12. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 734
  13. 2022Language-driven Semantic Segmentationcited by 717
  14. 2022MaPLe: Multi-modal Prompt Learningcited by 687
  15. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 673
  16. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 627
  17. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 614
  18. 2021ImageNet-21K Pretraining for the Massescited by 613
  19. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 532
  20. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  21. 2022Expanding Language-Image Pretrained Models for General Video Recognitioncited by 321
  22. 2022Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Modelcited by 317
  23. 2019Zero-Shot Semantic Segmentationcited by 245
  24. 2022PromptDet: Towards Open-vocabulary Detection using Uncurated Imagescited by 133
  25. 2021Probabilistic two-stage detectioncited by 130
  26. 2020Context-aware Feature Generation for Zero-shot Semantic Segmentationcited by 94
  27. 2020Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Modelscited by 90
  28. 2020Delving into Inter-Image Invariance for Unsupervised Visual Representationscited by 37
  29. 2021Learning Relation Alignment for Calibrated Cross-modal Retrievalcited by 21
  30. 2020EqCo: Equivalent Rules for Self-supervised Contrastive Learningcited by 12
  31. 2022Delving into the Openness of CLIPcited by 10
  32. 2022SPT: Semi-Parametric Prompt Tuning for Multitask Prompted Learningcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 14; 2024: 12; 2025: 7; 2026: 2.

35 papers.

Sort:

By area:

  1. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 455
  2. 2023Towards Open Vocabulary Learning: A Surveycited by 166
  3. 2024DeCo: Decoupling Token Compression from Semantic Abstraction in Multimodal Large Language Modelscited by 75
  4. 2024PromptKD: Unsupervised Prompt Distillation for Vision-Language Modelscited by 72
  5. 2023A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Futurecited by 61
  6. 2023Towards End-to-End Embodied Decision Making via Multi-modal Large Language Model: Explorations with GPT4-Vision and Beyondcited by 51
  7. 2023TESTA: Temporal-Spatial Token Aggregation for Long-form Video-Language Understandingcited by 49
  8. 2023Multi-modal Queried Object Detection in the Wildcited by 47
  9. 2024PCA-Bench: Evaluating Multimodal Large Language Models in Perception-Cognition-Action Chaincited by 29
  10. 2024AWT: Transferring Vision-Language Models via Augmentation, Weighting, and Transportationcited by 28
  11. 2023Link-Context Learning for Multimodal LLMscited by 24
  12. 2024Learning Background Prompts to Discover Implicit Knowledge for Open Vocabulary Object Detectioncited by 24
23 more
  1. 2024Improving Zero-shot Generalization of Learned Prompts via Unsupervised Knowledge Distillationcited by 18
  2. 2023Simple Image-level Classification Improves Open-vocabulary Object Detectioncited by 15
  3. 2024World to Code: Multi-modal Data Generation via Self-Instructed Compositional Captioning and Filteringcited by 15
  4. 2024SHiNe: Semantic Hierarchy Nexus for Open-vocabulary Object Detectioncited by 11
  5. 2023Auto-Vocabulary Semantic Segmentationcited by 8
  6. 2023OV-VG: A Benchmark for Open-Vocabulary Visual Groundingcited by 7
  7. 2023Rethinking Evaluation Metrics of Open-Vocabulary Segmentaioncited by 7
  8. 2023Retrieval-Enhanced Visual Prompt Learning for Few-shot Classificationcited by 6
  9. 2025Kaputt: A Large-Scale Dataset for Visual Defect Detectioncited by 6
  10. 2023DPL: Decoupled Prompt Learning for Vision-Language Modelscited by 4
  11. 2023Towards Realistic Zero-Shot Classification via Self Structural Semantic Alignmentcited by 4
  12. 2024Enhancing Fine-Grained Image Classifications via Cascaded Vision Language Modelscited by 4
  13. 2024Revisiting Prompt Pretraining of Vision-Language Modelscited by 4
  14. 2024Robust Adaptation of Foundation Models with Black-Box Visual Promptingcited by 3
  15. 2025Generalized Few-Shot Out-of-Distribution Detectioncited by 1
  16. 2025Constrained Prompt Enhancement for Improving Zero-Shot Generalization of Vision-Language Modelscited by 1
  17. 2025Beyond the Seen: Bounded Distribution Estimation for Open-Vocabulary Learningcited by 1
  18. 2025AnchorOPT: Towards Optimizing Dynamic Anchors for Adaptive Prompt Learningcited by 1
  19. 2026A Training-Free Guess What Vision Language Model from Snippets to Open-Vocabulary Object Detectioncited by 1
  20. 2024Enabling Generalized Zero-shot Learning Towards Unseen Domains by Intrinsic Learning from Redundant LLM Semanticscited by 0
  21. 2025State and Scene Enhanced Prototypes for Weakly Supervised Open-Vocabulary Object Detectioncited by 0
  22. 2025Open Vocabulary Compositional Explanations for Neuron Alignmentcited by 0
  23. 2026SpecPL: Disentangling Spectral Granularity for Prompt Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Prompting through Prototype: A Prototype-based Prompt Learning on Pretrained Vision-Language Models
  2. 2023PRE: Vision-Language Prompt Learning with Reparameterization Encoder
  3. 2024Meta-Prompting for Automating Zero-shot Visual Recognition with LLMs
  4. 2025Show or Tell? Effectively prompting Vision-Language Models for semantic segmentation
  5. 2024Adapting Vision-Language Models to Open Classes via Test-Time Prompt Tuning
  6. 2023CoPL: Contextual Prompt Learning for Vision-Language Understanding
  7. 2025ProAPO: Progressively Automatic Prompt Optimization for Visual Classification
  8. 2026AdaBoosting Text Prompts for Vision-Language Models