Data to 7 Oct 2026

Open-Vocabulary Object Detection Using Captions

arXiv 2011.10678 ↗First posted 20 Nov 2020cs.CV cs.AI cs.LGcited by 427

Not on the map. Why?

Cited by 66 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Despite the remarkable accuracy of deep neural networks in object detection, they are costly to train and scale due to supervision requirements. Particularly, learning more object categories typically requires proportionally more bounding box annotations. Weakly supervised and zero-shot learning techniques have been explored to scale object detectors to more categories with less supervision, but they have not been as successful and widely adopted as supervised models. In this paper, we put forth a novel formulation of the object detection problem, namely open-vocabulary object detection, which is more general, more practical, and more effective than weakly supervised and zero-shot approaches. We propose a new method to train object detectors using bounding box annotations for a limited set of object categories, as well as image-caption pairs that cover a larger variety of objects at a significantly lower cost. We show that the proposed method can detect and localize objects for which no bounding box annotation is provided during training, at a significantly higher accuracy than zero-shot approaches. Meanwhile, objects with bounding box annotation can be detected almost as accurately as supervised methods, which is significantly better than weakly supervised baselines. Accordingly, we establish a new state of the art for scalable object detection.

Key facts. Cited by 427 papers in the map (data through 7 Oct 2026). First posted on arXiv 20 Nov 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 14; 2022: 47; 2023: 129; 2024: 104; 2025: 81; 2026: 52.

427 papers.

Sort:

By area:

2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
2021Grounded Language-Image Pre-trainingcited by 1,147
2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 774
2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 613
2021RegionCLIP: Region-based Language-Image Pretrainingcited by 612
2025Visual-RFT: Visual Reinforcement Fine-Tuningcited by 501
2024YOLO-World: Real-Time Open-Vocabulary Object Detectioncited by 498
2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 471
2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Modelscited by 371
2022Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Modelcited by 317
2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 303

38 more

2022GLIPv2: Unifying Localization and Vision-Language Understandingcited by 275
2022OpenOOD: Benchmarking Generalized Out-of-Distribution Detectioncited by 267
2023A Cookbook of Self-Supervised Learningcited by 248
2024EMMA: End-to-End Multimodal Model for Autonomous Drivingcited by 226
2022Open-Vocabulary DETR with Conditional Matchingcited by 220
2023Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIPcited by 199
2021Decoupling Zero-Shot Semantic Segmentationcited by 198
2023A Simple Framework for Open-Vocabulary Segmentation and Detectioncited by 197
2022PLA: Language-Driven Open-Vocabulary 3D Scene Understandingcited by 176
2023Towards Open Vocabulary Learning: A Surveycited by 165
2022F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Modelscited by 149
2022Bridging the Gap between Object and Image-level Representations for Open-Vocabulary Detectioncited by 145
2022ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Modelscited by 141
2022PromptDet: Towards Open-vocabulary Detection using Uncurated Imagescited by 133
2023CORA: Adapting CLIP for Open-Vocabulary Detection with Region Prompting and Anchor Pre-Matchingcited by 133
2023Hyperbolic Image-Text Representationscited by 132
2022Open-Vocabulary Universal Image Segmentation with MaskCLIPcited by 131
2023Transformer-Based Visual Segmentation: A Surveycited by 124
2022GRiT: A Generative Region-to-text Transformer for Object Understandingcited by 119
2023Aligning Bag of Regions for Open-Vocabulary Object Detectioncited by 119
2023CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Predictioncited by 114
2024Grounded 3D-LLM with Referent Tokenscited by 111
2022VLT: Vision-Language Transformer and Query Generation for Referring Segmentationcited by 110
2023Contextual Object Detection with Multimodal Large Language Modelscited by 100
2022Exploiting Unlabeled Data with Vision and Language Models for Object Detectioncited by 97
2022Learning Object-Language Alignments for Open-Vocabulary Object Detectioncited by 95
2023Referring Multi-Object Trackingcited by 91
2023Region-Aware Pretraining for Open-Vocabulary Object Detection with Vision Transformerscited by 91
2023Language Embedded Radiance Fields for Zero-Shot Task-Oriented Graspingcited by 91
2023CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIPcited by 89
2023CLIP the Gap: A Single Domain Generalization Approach for Object Detectioncited by 87
2021Class-agnostic Object Detection with Multi-modal Transformercited by 85
2021Open-Vocabulary Instance Segmentation via Robust Cross-Modal Pseudo-Labelingcited by 84
2023Open-Vocabulary Point-Cloud Object Detection without 3D Annotationcited by 81
2023V3Det: Vast Vocabulary Visual Detection Datasetcited by 81
2025FG-CLIP: Fine-Grained Visual and Textual Alignmentcited by 78
2022SIGMA: Semantic-complete Graph Matching for Domain Adaptive Object Detectioncited by 74
2024Open-Vocabulary SAM: Segment and Recognize Twenty-thousand Classes Interactivelycited by 74

50 most cited of 427 All 427 (JSON)