# What Makes Good Open-Vocabulary Detector: A Disassembling Perspective arXiv: https://arxiv.org/abs/2309.00227 ยท HTML: https://mlsanity.com/paper/2309.00227/ ## Key facts - Cited by 2 papers in the map (data through 14 Oct 2026). - Builds on 25 of its 28 references that are in the map. - First posted on arXiv 1 Sep 2023. ## Builds on (most cited first, 25 of 25) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014) - [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015) - [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014) - [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015) - [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021) - [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021) - [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019) - [Open-vocabulary Object Detection via Vision and Language Knowledge Distillation](https://mlsanity.com/paper/2104.13921/) (2021) - [RegionCLIP: Region-based Language-Image Pretraining](https://mlsanity.com/paper/2112.09106/) (2021) - [Detecting Twenty-thousand Classes using Image-level Supervision](https://mlsanity.com/paper/2201.02605/) (2022) - [Open-Vocabulary Object Detection Using Captions](https://mlsanity.com/paper/2011.10678/) (2020) - [Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Model](https://mlsanity.com/paper/2203.14940/) (2022) - [F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Models](https://mlsanity.com/paper/2209.15639/) (2022) - [Bridging the Gap between Object and Image-level Representations for Open-Vocabulary Detection](https://mlsanity.com/paper/2207.03482/) (2022) - [CORA: Adapting CLIP for Open-Vocabulary Detection with Region Prompting and Anchor Pre-Matching](https://mlsanity.com/paper/2303.13076/) (2023) - [Aligning Bag of Regions for Open-Vocabulary Object Detection](https://mlsanity.com/paper/2302.13996/) (2023) - [Learning Open-World Object Proposals without Learning to Classify](https://mlsanity.com/paper/2108.06753/) (2021) - [Chinese CLIP: Contrastive Vision-Language Pretraining in Chinese](https://mlsanity.com/paper/2211.01335/) (2022) - [Exploiting Unlabeled Data with Vision and Language Models for Object Detection](https://mlsanity.com/paper/2207.08954/) (2022) - [Open Vocabulary Object Detection with Pseudo Bounding-Box Labels](https://mlsanity.com/paper/2111.09452/) (2021) - [Open-Vocabulary One-Stage Detection with Hierarchical Visual-Language Knowledge Distillation](https://mlsanity.com/paper/2203.10593/) (2022) - [Detecting Everything in the Open World: Towards Universal Object Detection](https://mlsanity.com/paper/2303.11749/) (2023) - [BigDetection: A Large-scale Benchmark for Improved Object Detector Pre-training](https://mlsanity.com/paper/2203.13249/) (2022) - [Open Vocabulary Object Detection with Proposal Mining and Prediction Equalization](https://mlsanity.com/paper/2206.11134/) (2022) ## Built on by (most cited first, 2 of 2) - [FG-CLIP: Fine-Grained Visual and Textual Alignment](https://mlsanity.com/paper/2505.05071/) (2025) - [IAA: Inner-Adaptor Architecture Empowers Frozen Large Language Model with Multimodal Capabilities](https://mlsanity.com/paper/2408.12902/) (2024) ## Abstract (arXiv) Open-vocabulary detection (OVD) is a new object detection paradigm, aiming to localize and recognize unseen objects defined by an unbounded vocabulary. This is challenging since traditional detectors can only learn from pre-defined categories and thus fail to detect and localize objects out of pre-defined vocabulary. To handle the challenge, OVD leverages pre-trained cross-modal VLM, such as CLIP, ALIGN, etc. Previous works mainly focus on the open vocabulary classification part, with less attention on the localization part. We argue that for a good OVD detector, both classification and localization should be parallelly studied for the novel object categories. We show in this work that improving localization as well as cross-modal classification complement each other, and compose a good OVD detector jointly. We analyze three families of OVD methods with different design emphases. We first propose a vanilla method,i.e., cropping a bounding box obtained by a localizer and resizing it into the CLIP. We next introduce another approach, which combines a standard two-stage object detector with CLIP. A two-stage object detector includes a visual backbone, a region proposal network (RPN), and a region of interest (RoI) head. We decouple RPN and ROI head (DRR) and use RoIAlign to extract meaningful features. In this case, it avoids resizing objects. To further accelerate the training time and reduce the model parameters, we couple RPN and ROI head (CRR) as the third approach. We conduct extensive experiments on these three types of approaches in different settings. On the OVD-COCO benchmark, DRR obtains the best performance and achieves 35.8 Novel AP50, an absolute 2.8 gain over the previous state-of-the-art (SOTA). For OVD-LVIS, DRR surpasses the previous SOTA by 1.9 AP50 in rare categories. We also provide an object detection dataset called PID and provide a baseline on PID.