mlsanityData to 7 Oct 2026

Learning Object-Language Alignments for Open-Vocabulary Object Detection

arXiv 2211.14843 ↗First posted 27 Nov 2022cs.CVcited by 95

Not on the map. Why?

Cited by 20 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Existing object detection methods are bounded in a fixed-set vocabulary by costly labeled data. When dealing with novel categories, the model has to be retrained with more bounding box annotations. Natural language supervision is an attractive alternative for its annotation-free attributes and broader object concepts. However, learning open-vocabulary object detection from language is challenging since image-text pairs do not contain fine-grained object-language alignments. Previous solutions rely on either expensive grounding annotations or distilling classification-oriented vision models. In this paper, we propose a novel open-vocabulary object detection framework directly learning from image-text pair data. We formulate object-language alignment as a set matching problem between a set of image region features and a set of word embeddings. It enables us to train an open-vocabulary object detector on image-text pairs in a much simple and effective way. Extensive experiments on two benchmark datasets, COCO and LVIS, demonstrate our superior performance over the competing approaches on novel categories, e.g. achieving 32.0% mAP on COCO and 21.7% mask mAP on LVIS. Code is available at: https://github.com/clin1223/VLDet.

Key facts. Cited by 95 papers in the map (data through 7 Oct 2026). Builds on 52 of its 63 references that are in the map. First posted on arXiv 27 Nov 2022.

Builds on

52 of 63 references.

References · 52

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  5. 2020End-to-End Object Detection with Transformerscited by 7,384
  6. 2017Mask R-CNNcited by 7,259
  7. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  9. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  10. 2015Fast R-CNNcited by 2,930
  11. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  12. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,277
38 more
  1. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  2. 2016YOLO9000: Better, Faster, Strongercited by 1,429
  3. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  4. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,306
  5. 2017Cascade R-CNN: Delving into High Quality Object Detectioncited by 1,275
  6. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  7. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  8. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  9. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  10. 2021Grounded Language-Image Pre-trainingcited by 1,147
  11. 2016Modeling Context in Referring Expressionscited by 1,054
  12. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  13. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  14. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  15. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  16. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 613
  17. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 612
  18. 2020Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentationcited by 595
  19. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  20. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 530
  21. 2020Sparse R-CNN: End-to-End Object Detection with Learnable Proposalscited by 494
  22. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 471
  23. 2020Open-Vocabulary Object Detection Using Captionscited by 427
  24. 2021Conditional DETR for Fast Training Convergencecited by 422
  25. 2017Semantic Foggy Scene Understanding with Synthetic Datacited by 401
  26. 2022Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Modelcited by 317
  27. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 303
  28. 2016Driving in the Matrix: Can Virtual Worlds Replace Human-Generated Annotations for Real World Tasks?cited by 194
  29. 2018Zero-Shot Object Detectioncited by 189
  30. 2015Weakly Supervised Deep Detection Networkscited by 183
  31. 2021Instances as Queriescited by 157
  32. 2021Probabilistic two-stage detectioncited by 130
  33. 2017Multiple Instance Detection Network with Online Instance Classifier Refinementcited by 117
  34. 2022Exploiting Unlabeled Data with Vision and Language Models for Object Detectioncited by 97
  35. 2021Open-Vocabulary Instance Segmentation via Robust Cross-Modal Pseudo-Labelingcited by 84
  36. 2020What Makes for End-to-End Object Detection?cited by 47
  37. 2022X-DETR: A Versatile Architecture for Instance-wise Vision-Language Taskscited by 41
  38. 2015Weakly Supervised Object Localization with Multi-fold Multiple Instance Learningcited by 39

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 30; 2024: 35; 2025: 19; 2026: 11.

95 papers.

Sort:

By area:

  1. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  2. 2024YOLO-World: Real-Time Open-Vocabulary Object Detectioncited by 498
  3. 2023Towards Open Vocabulary Learning: A Surveycited by 165
  4. 2023Aligning Bag of Regions for Open-Vocabulary Object Detectioncited by 119
  5. 2023CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Predictioncited by 114
  6. 2025FG-CLIP: Fine-Grained Visual and Textual Alignmentcited by 78
  7. 2023General Object Foundation Model for Images and Videos at Scalecited by 71
  8. 2023Zero-shot Referring Image Segmentation with Global-Local Context Featurescited by 61
  9. 2023A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Futurecited by 59
  10. 2023Going Denser with Open-Vocabulary Part Segmentationcited by 51
  11. 2024DetCLIPv3: Towards Versatile Generative Open-vocabulary Object Detectioncited by 48
  12. 2023CoDet: Co-Occurrence Guided Region-Word Alignment for Open-Vocabulary Object Detectioncited by 46
38 more
  1. 2023EdaDet: Open-Vocabulary Object Detection Using Early Dense Alignmentcited by 44
  2. 2024LLMs Meet VLMs: Boost Open Vocabulary Object Detection with Fine-grained Descriptorscited by 39
  3. 2024ViTamin: Designing Scalable Vision Models in the Vision-Language Eracited by 39
  4. 2023OV-PARTS: Towards Open-Vocabulary Part Segmentationcited by 31
  5. 2023LP-OVOD: Open-Vocabulary Object Detection by Linear Probingcited by 29
  6. 2024Generative Region-Language Pretraining for Open-Ended Object Detectioncited by 29
  7. 2024OV-DINO: Unified Open-Vocabulary Detection with Language-Aware Selective Fusioncited by 28
  8. 2024InstaGen: Enhancing Object Detection by Training on Synthetic Datasetcited by 26
  9. 2024AIDE: An Automatic Data Engine for Object Detection in Autonomous Drivingcited by 26
  10. 2023Taming Self-Training for Open-Vocabulary Object Detectioncited by 25
  11. 2023Object2Scene: Putting Objects in Context for Open-Vocabulary 3D Detectioncited by 24
  12. 2024Learning Background Prompts to Discover Implicit Knowledge for Open Vocabulary Object Detectioncited by 24
  13. 2024OV-DQUO: Open-Vocabulary DETR with Denoising Text Query Training and Open-World Unknown Objects Supervisioncited by 22
  14. 2023Text Augmented Spatial-aware Zero-shot Referring Image Segmentationcited by 21
  15. 2023CLIM: Contrastive Language-Image Mosaic for Region Representationcited by 21
  16. 2024Retrieval-Augmented Open-Vocabulary Object Detectioncited by 20
  17. 2024RoboUniView: Visual-Language Model with Unified View Representation for Robotic Manipulationcited by 20
  18. 2023OmniLabel: A Challenging Benchmark for Language-Based Object Detectioncited by 16
  19. 2023ProxyDet: Synthesizing Proxy Novel Classes via Classwise Mixup for Open-Vocabulary Object Detectioncited by 15
  20. 2024OpenPSG: Open-set Panoptic Scene Graph Generation via Large Multimodal Modelscited by 15
  21. 2024PartGLEE: A Foundation Model for Recognizing and Parsing Any Objectscited by 15
  22. 2025Conformal Prediction for Zero-Shot Modelscited by 14
  23. 2026Vision Transformers Need More Than Registerscited by 14
  24. 2023Open-Vocabulary Object Detection using Pseudo Caption Labelscited by 13
  25. 2024Open-Set Recognition in the Age of Vision-Language Modelscited by 13
  26. 2024Frozen-DETR: Enhancing DETR with Image Understanding from Frozen Foundation Modelscited by 13
  27. 2025Refining CLIP's Spatial Awareness: A Visual-Centric Perspectivecited by 12
  28. 2023Generating Enhanced Negatives for Training Language-Based Object Detectorscited by 11
  29. 2024DiPEx: Dispersing Prompt Expansion for Class-Agnostic Object Detectioncited by 11
  30. 2024MarvelOVD: Marrying Object Recognition and Vision-Language Models for Robust Open-Vocabulary Object Detectioncited by 11
  31. 2025OVTR: End-to-End Open-Vocabulary Multiple Object Tracking with Transformercited by 11
  32. 2023Zero-shot Nuclei Detection via Visual-Language Pre-trained Modelscited by 10
  33. 2025Vision-Language Model for Object Detection and Segmentation: A Review and Evaluationcited by 10
  34. 2023ViLLA: Fine-Grained Vision-Language Representation Learning from Real-World Datacited by 9
  35. 2024Towards Open-World Mobile Manipulation in Homes: Lessons from the Neurips 2023 HomeRobot Open Vocabulary Mobile Manipulation Challengecited by 9
  36. 2024Advancing Molecular Graph-Text Pre-training via Fine-grained Alignmentcited by 9
  37. 2024Unbiased Region-Language Alignment for Open-Vocabulary Dense Predictioncited by 9
  38. 2025A Hierarchical Semantic Distillation Framework for Open-Vocabulary Object Detectioncited by 9

50 most cited of 95 All 95 (JSON)