mlsanityData to 7 Oct 2026

Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Model

arXiv 2203.14940 ↗First posted 28 Mar 2022cs.CVcited by 317

On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection

Abstract (arXiv)

Recently, vision-language pre-training shows great potential in open-vocabulary object detection, where detectors trained on base classes are devised for detecting new classes. The class text embedding is firstly generated by feeding prompts to the text encoder of a pre-trained vision-language model. It is then used as the region classifier to supervise the training of a detector. The key element that leads to the success of this model is the proper prompt, which requires careful words tuning and ingenious design. To avoid laborious prompt engineering, there are some prompt representation learning methods being proposed for the image classification task, which however can only be sub-optimal solutions when applied to the detection task. In this paper, we introduce a novel method, detection prompt (DetPro), to learn continuous prompt representations for open-vocabulary object detection based on the pre-trained vision-language model. Different from the previous classification-oriented methods, DetPro has two highlights: 1) a background interpretation scheme to include the proposals in image background into the prompt training; 2) a context grading scheme to separate proposals in image foreground for tailored prompt training. We assemble DetPro with ViLD, a recent state-of-the-art open-world object detector, and conduct experiments on the LVIS as well as transfer learning on the Pascal VOC, COCO, Objects365 datasets. Experimental results show that our DetPro outperforms the baseline ViLD in all settings, e.g., +3.4 APbox and +3.0 APmask improvements on the novel classes of LVIS. Code and models are available at https://github.com/dyabel/detpro.

Key facts. Cited by 317 papers in the map (data through 7 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 29 of its 39 references that are in the map. First posted on arXiv 28 Mar 2022.

Builds on

29 of 39 references.

References · 29

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  6. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  8. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  9. 2020Supervised Contrastive Learningcited by 2,803
  10. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  11. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  12. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
17 more
  1. 2017Learning to Compare: Relation Network for Few-Shot Learningcited by 1,139
  2. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  3. 2017Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Uglycited by 536
  4. 2020Open-Vocabulary Object Detection Using Captionscited by 427
  5. 2021End-to-End Semi-Supervised Object Detection with Soft Teachercited by 315
  6. 2018Recent Advances in Open Set Recognition: A Surveycited by 309
  7. 2021CLIP2Video: Mastering Video-Text Retrieval via Image CLIPcited by 223
  8. 2018Zero-Shot Object Detectioncited by 189
  9. 2020Overcoming Classifier Imbalance for Long-tail Object Detection with Balanced Group Softmaxcited by 152
  10. 2018Out-of-Distribution Detection Using an Ensemble of Self Supervised Leave-out Classifierscited by 126
  11. 2021Aligning Pretraining for Detection via Object-Level Contrastive Learningcited by 110
  12. 2020Point-Set Anchors for Object Detection, Instance Segmentation and Pose Estimationcited by 57
  13. 2020Global Context Networkscited by 35
  14. 2020RelationNet++: Bridging Visual Representations for Object Detection via Transformer Decodercited by 31
  15. 2020Restoring Negative Information in Few-Shot Object Detectioncited by 28
  16. 2018Zero-Shot Detectioncited by 27
  17. 2021Bootstrap Your Object Detector via Mixed Trainingcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 24; 2023: 105; 2024: 87; 2025: 65; 2026: 36.

317 papers.

Sort:

By area:

  1. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  2. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 560
  3. 2024YOLO-World: Real-Time Open-Vocabulary Object Detectioncited by 498
  4. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Modelscited by 379
  5. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Modelscited by 371
  6. 2022DetCLIP: Dictionary-Enriched Visual-Concept Paralleled Pre-training for Open-world Detectioncited by 176
  7. 2022PLA: Language-Driven Open-Vocabulary 3D Scene Understandingcited by 176
  8. 2023Towards Open Vocabulary Learning: A Surveycited by 165
  9. 2022F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Modelscited by 149
  10. 2022Bridging the Gap between Object and Image-level Representations for Open-Vocabulary Detectioncited by 145
  11. 2022Unified Vision and Language Prompt Learningcited by 145
  12. 2025Reinforcing Spatial Reasoning in Vision-Language Models with Interwoven Thinking and Visual Drawingcited by 130
38 more
  1. 2022Unsupervised Prompt Learning for Vision-Language Modelscited by 125
  2. 2023Aligning Bag of Regions for Open-Vocabulary Object Detectioncited by 119
  3. 2022SuS-X: Training-Free Name-Only Transfer of Vision-Language Modelscited by 118
  4. 2023CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Predictioncited by 114
  5. 2023LoCoOp: Few-Shot Out-of-Distribution Detection via Prompt Learningcited by 107
  6. 2022CALIP: Zero-Shot Enhancement of CLIP with Parameter-free Attentioncited by 103
  7. 2023Contextual Object Detection with Multimodal Large Language Modelscited by 100
  8. 2022Learning Object-Language Alignments for Open-Vocabulary Object Detectioncited by 95
  9. 2023RegionPLC: Regional Point-Language Contrastive Learning for Open-World 3D Scene Understandingcited by 93
  10. 2023Vision-Language Models in Remote Sensing: Current Progress and Future Trendscited by 93
  11. 2023Region-Aware Pretraining for Open-Vocabulary Object Detection with Vision Transformerscited by 91
  12. 2023Vita-CLIP: Video and text adaptive CLIP via Multimodal Promptingcited by 90
  13. 2023CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIPcited by 89
  14. 2024Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunitiescited by 83
  15. 2023V3Det: Vast Vocabulary Visual Detection Datasetcited by 81
  16. 2023Fine-Grained Visual Promptingcited by 81
  17. 2023Efficient Emotional Adaptation for Audio-Driven Talking-Head Generationcited by 81
  18. 2023Object-Aware Distillation Pyramid for Open-Vocabulary Object Detectioncited by 63
  19. 2023Detecting Everything in the Open World: Towards Universal Object Detectioncited by 62
  20. 2023Zero-shot Referring Image Segmentation with Global-Local Context Featurescited by 61
  21. 2023HOICLIP: Efficient Knowledge Transfer for HOI Detection with Vision-Language Modelscited by 60
  22. 2023A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Futurecited by 59
  23. 2023OVTrack: Open-Vocabulary Multiple Object Trackingcited by 53
  24. 2022Bayesian Prompt Learning for Image-Language Model Generalizationcited by 51
  25. 2023Going Denser with Open-Vocabulary Part Segmentationcited by 51
  26. 2023CoDA: Collaborative Novel Box Discovery and Cross-modal Alignment for Open-vocabulary 3D Object Detectioncited by 51
  27. 2022X-Paste: Revisiting Scalable Copy-Paste for Instance Segmentation using CLIP and StableDiffusioncited by 49
  28. 2024DetCLIPv3: Towards Versatile Generative Open-vocabulary Object Detectioncited by 48
  29. 2023Multi-modal Queried Object Detection in the Wildcited by 47
  30. 2023Natural Language-Assisted Sign Language Recognitioncited by 46
  31. 2023CoDet: Co-Occurrence Guided Region-Word Alignment for Open-Vocabulary Object Detectioncited by 46
  32. 2023Towards Open-Vocabulary Video Instance Segmentationcited by 45
  33. 2023EdaDet: Open-Vocabulary Object Detection Using Early Dense Alignmentcited by 44
  34. 2022Understanding and Mitigating Overfitting in Prompt Tuning for Vision-Language Modelscited by 40
  35. 2025IS-Bench: Evaluating Interactive Safety of VLM-Driven Embodied Agents in Daily Household Taskscited by 40
  36. 2024LLMs Meet VLMs: Boost Open Vocabulary Object Detection with Fine-grained Descriptorscited by 39
  37. 2024ViTamin: Designing Scalable Vision Models in the Vision-Language Eracited by 39
  38. 2024Learning to Prompt with Text Only Supervision for Vision-Language Modelscited by 37

50 most cited of 317 All 317 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Fine-grained Visual-Text Prompt-Driven Self-Training for Open-Vocabulary Object Detection
  2. 2022Open-Vocabulary DETR with Conditional Matching
  3. 2023DetCLIPv2: Scalable Open-Vocabulary Object Detection Pre-training via Word-Region Alignment
  4. 2024DiPEx: Dispersing Prompt Expansion for Class-Agnostic Object Detection
  5. 2023CoPL: Contextual Prompt Learning for Vision-Language Understanding
  6. 2023PRE: Vision-Language Prompt Learning with Reparameterization Encoder
  7. 2024Adapting Vision-Language Models to Open Classes via Test-Time Prompt Tuning
  8. 2024OVLW-DETR: Open-Vocabulary Light-Weighted Detection Transformer