# Open Vocabulary Object Detection with Pseudo Bounding-Box Labels

arXiv: https://arxiv.org/abs/2111.09452 · HTML: https://mlsanity.com/paper/2111.09452/

## Key facts

- Cited by 71 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 36 references that are in the map.
- First posted on arXiv 18 Nov 2021.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://mlsanity.com/paper/1908.03557/) (2019)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [Center-based 3D Object Detection and Tracking](https://mlsanity.com/paper/2006.11275/) (2020)
- [Sparse R-CNN: End-to-End Object Detection with Learnable Proposals](https://mlsanity.com/paper/2011.12450/) (2020)
- [Open-Vocabulary Object Detection Using Captions](https://mlsanity.com/paper/2011.10678/) (2020)
- [Towards Open World Object Detection](https://mlsanity.com/paper/2103.02603/) (2021)
- [DetCo: Unsupervised Contrastive Learning for Object Detection](https://mlsanity.com/paper/2102.04803/) (2021)
- [Zero-Shot Object Detection](https://mlsanity.com/paper/1804.04340/) (2018)
- [Weakly Supervised Deep Detection Networks](https://mlsanity.com/paper/1511.02853/) (2015)
- [PCL: Proposal Cluster Learning for Weakly Supervised Object Detection](https://mlsanity.com/paper/1807.03342/) (2018)
- [Dont Even Look Once: Synthesizing Features for Zero-Shot Detection](https://mlsanity.com/paper/1911.07933/) (2019)
- [Cap2Det: Learning to Amplify Weak Caption Supervision for Object Detection](https://mlsanity.com/paper/1907.10164/) (2019)
- [Zero-Shot Detection](https://mlsanity.com/paper/1803.07113/) (2018)
- [C-WSL: Count-guided Weakly Supervised Localization](https://mlsanity.com/paper/1711.05282/) (2017)

## Built on by (most cited first, 30 of 71)

- [Vision-Language Models for Vision Tasks: A Survey](https://mlsanity.com/paper/2304.00685/) (2023)
- [ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding](https://mlsanity.com/paper/2212.05171/) (2022)
- [A Simple Framework for Open-Vocabulary Segmentation and Detection](https://mlsanity.com/paper/2303.08131/) (2023)
- [Towards Open Vocabulary Learning: A Survey](https://mlsanity.com/paper/2306.15880/) (2023)
- [CORA: Adapting CLIP for Open-Vocabulary Detection with Region Prompting and Anchor Pre-Matching](https://mlsanity.com/paper/2303.13076/) (2023)
- [GRiT: A Generative Region-to-text Transformer for Object Understanding](https://mlsanity.com/paper/2212.00280/) (2022)
- [Aligning Bag of Regions for Open-Vocabulary Object Detection](https://mlsanity.com/paper/2302.13996/) (2023)
- [CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Prediction](https://mlsanity.com/paper/2310.01403/) (2023)
- [V3Det: Vast Vocabulary Visual Detection Dataset](https://mlsanity.com/paper/2304.03752/) (2023)
- [Object-Aware Distillation Pyramid for Open-Vocabulary Object Detection](https://mlsanity.com/paper/2303.05892/) (2023)
- [A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Future](https://mlsanity.com/paper/2307.09220/) (2023)
- [DetCLIPv3: Towards Versatile Generative Open-vocabulary Object Detection](https://mlsanity.com/paper/2404.09216/) (2024)
- [CoDet: Co-Occurrence Guided Region-Word Alignment for Open-Vocabulary Object Detection](https://mlsanity.com/paper/2310.16667/) (2023)
- [OpenESS: Event-based Semantic Scene Understanding with Open Vocabularies](https://mlsanity.com/paper/2405.05259/) (2024)
- [MosaicFusion: Diffusion Models as Data Augmenters for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/2309.13042/) (2023)
- [Open-vocabulary Attribute Detection](https://mlsanity.com/paper/2211.12914/) (2022)
- [LP-OVOD: Open-Vocabulary Object Detection by Linear Probing](https://mlsanity.com/paper/2310.17109/) (2023)
- [Zero-shot Visual Relation Detection via Composite Visual Cues from Large Language Models](https://mlsanity.com/paper/2305.12476/) (2023)
- [InstaGen: Enhancing Object Detection by Training on Synthetic Dataset](https://mlsanity.com/paper/2402.05937/) (2024)
- [Taming Self-Training for Open-Vocabulary Object Detection](https://mlsanity.com/paper/2308.06412/) (2023)
- [Learning Background Prompts to Discover Implicit Knowledge for Open Vocabulary Object Detection](https://mlsanity.com/paper/2406.00510/) (2024)
- [Learning to Detect and Segment for Open Vocabulary Object Detection](https://mlsanity.com/paper/2212.12130/) (2022)
- [CLIM: Contrastive Language-Image Mosaic for Region Representation](https://mlsanity.com/paper/2312.11376/) (2023)
- [Retrieval-Augmented Open-Vocabulary Object Detection](https://mlsanity.com/paper/2404.05687/) (2024)
- [ImOV3D: Learning Open-Vocabulary Point Clouds 3D Object Detection from Only 2D Images](https://mlsanity.com/paper/2410.24001/) (2024)
- [Mask-free OVIS: Open-Vocabulary Instance Segmentation without Manual Mask Annotations](https://mlsanity.com/paper/2303.16891/) (2023)
- [Scene Graph Generation with Role-Playing Large Language Models](https://mlsanity.com/paper/2410.15364/) (2024)
- [ProxyDet: Synthesizing Proxy Novel Classes via Classwise Mixup for Open-Vocabulary Object Detection](https://mlsanity.com/paper/2312.07266/) (2023)
- [Open-Vocabulary Object Detection using Pseudo Caption Labels](https://mlsanity.com/paper/2303.13040/) (2023)
- [Open World Object Detection: A Survey](https://mlsanity.com/paper/2410.11301/) (2024)

## Abstract (arXiv)

Despite great progress in object detection, most existing methods work only on a limited set of object categories, due to the tremendous human effort needed for bounding-box annotations of training data. To alleviate the problem, recent open vocabulary and zero-shot detection methods attempt to detect novel object categories beyond those seen during training. They achieve this goal by training on a pre-defined base categories to induce generalization to novel objects. However, their potential is still constrained by the small set of base categories available for training. To enlarge the set of base classes, we propose a method to automatically generate pseudo bounding-box annotations of diverse objects from large-scale image-caption pairs. Our method leverages the localization ability of pre-trained vision-language models to generate pseudo bounding-box labels and then directly uses them for training object detectors. Experimental results show that our method outperforms the state-of-the-art open vocabulary detector by 8% AP on COCO novel categories, by 6.3% AP on PASCAL VOC, by 2.3% AP on Objects365 and by 2.8% AP on LVIS. Code is available at https://github.com/salesforce/PB-OVD.