mlsanityData to 14 Oct 2026

Aligning Bag of Regions for Open-Vocabulary Object Detection

arXiv 2302.13996 ↗First posted 27 Feb 2023cs.CVcited by 119

On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection

Abstract (arXiv)

Pre-trained vision-language models (VLMs) learn to align vision and language representations on large-scale datasets, where each image-text pair usually contains a bag of semantic concepts. However, existing open-vocabulary object detectors only align region embeddings individually with the corresponding features extracted from the VLMs. Such a design leaves the compositional structure of semantic concepts in a scene under-exploited, although the structure may be implicitly learned by the VLMs. In this work, we propose to align the embedding of bag of regions beyond individual regions. The proposed method groups contextually interrelated regions as a bag. The embeddings of regions in a bag are treated as embeddings of words in a sentence, and they are sent to the text encoder of a VLM to obtain the bag-of-regions embedding, which is learned to be aligned to the corresponding features extracted by a frozen VLM. Applied to the commonly used Faster R-CNN, our approach surpasses the previous best results by 4.6 box AP50 and 2.8 mask AP on novel categories of open-vocabulary COCO and LVIS benchmarks, respectively. Code and models are available at https://github.com/wusize/ovdet.

Key facts. Cited by 119 papers in the map (data through 14 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 51 of its 62 references that are in the map. First posted on arXiv 27 Feb 2023.

Builds on

51 of 62 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Model28 Mar

Other references · 50

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  8. 2020End-to-End Object Detection with Transformerscited by 7,414
  9. 2017Mask R-CNNcited by 7,277
  10. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
  11. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  12. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
38 more
  1. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
  2. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  3. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  4. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  5. 2019MMDetection: Open MMLab Detection Toolbox and Benchmarkcited by 1,385
  6. 2019Contrastive Multiview Codingcited by 1,301
  7. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  8. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,217
  9. 2021Grounded Language-Image Pre-trainingcited by 1,156
  10. 2017Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networkscited by 939
  11. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 906
  12. 2018Panoptic Segmentationcited by 802
  13. 2022Language-driven Semantic Segmentationcited by 717
  14. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 616
  15. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
  16. 2020Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentationcited by 596
  17. 2021Extract Free Dense Labels from CLIPcited by 583
  18. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
  19. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 532
  20. 2019Hybrid Task Cascade for Instance Segmentationcited by 481
  21. 2020Open-Vocabulary Object Detection Using Captionscited by 427
  22. 2018Deep Learning for Generic Object Detection: A Surveycited by 392
  23. 2022Open-Vocabulary DETR with Conditional Matchingcited by 220
  24. 2022Unified Contrastive Learning in Image-Text-Label Spacecited by 194
  25. 2018Zero-Shot Object Detectioncited by 189
  26. 2022Bridging the Gap between Object and Image-level Representations for Open-Vocabulary Detectioncited by 146
  27. 2013Zero-Shot Learning by Convex Combination of Semantic Embeddingscited by 135
  28. 2021Aligning Pretraining for Detection via Object-Level Contrastive Learningcited by 110
  29. 2022Learning Object-Language Alignments for Open-Vocabulary Object Detectioncited by 96
  30. 2021Class-agnostic Object Detection with Multi-modal Transformercited by 85
  31. 2021Open Vocabulary Object Detection with Pseudo Bounding-Box Labelscited by 71
  32. 2017MegDet: A Large Mini-Batch Object Detectorcited by 66
  33. 2022ProposalCLIP: Unsupervised Open-Category Object Proposal Generation via Exploiting CLIP Cuescited by 42
  34. 2014Zero Shot Recognition with Unreliable Attributescited by 39
  35. 2018Zero-Shot Object Detection by Hybrid Region Embeddingcited by 32
  36. 2020Synthesizing the Unseen for Zero-shot Object Detectioncited by 32
  37. 2020Background Learnable Cascade for Zero-Shot Object Detectioncited by 31
  38. 2021A Fistful of Words: Learning Transferable Visual Models from Bag-of-Words Supervisioncited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 28; 2024: 44; 2025: 28; 2026: 19.

119 papers.

Sort:

By area:

  1. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  2. 2024YOLO-World: Real-Time Open-Vocabulary Object Detectioncited by 503
  3. 2023AnomalyCLIP: Object-agnostic Prompt Learning for Zero-shot Anomaly Detectioncited by 271
  4. 2023Towards Open Vocabulary Learning: A Surveycited by 166
  5. 2023CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Predictioncited by 118
  6. 2023Contextual Object Detection with Multimodal Large Language Modelscited by 100
  7. 2024Open-Vocabulary SAM: Segment and Recognize Twenty-thousand Classes Interactivelycited by 75
  8. 2023SATR: Zero-Shot Semantic Segmentation of 3D Shapescited by 62
  9. 2023A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Futurecited by 61
  10. 2025LLMDet: Learning Strong Open-Vocabulary Object Detectors under the Supervision of Large Language Modelscited by 57
  11. 2023Going Denser with Open-Vocabulary Part Segmentationcited by 51
  12. 2023CoDet: Co-Occurrence Guided Region-Word Alignment for Open-Vocabulary Object Detectioncited by 46
38 more
  1. 2024DetToolChain: A New Prompting Paradigm to Unleash Detection Ability of MLLMcited by 46
  2. 2024LLMs Meet VLMs: Boost Open Vocabulary Object Detection with Fine-grained Descriptorscited by 40
  3. 2024ViTamin: Designing Scalable Vision Models in the Vision-Language Eracited by 39
  4. 2024OpenESS: Event-based Semantic Scene Understanding with Open Vocabulariescited by 38
  5. 2025DeCLIP: Decoupled Learning for Open-Vocabulary Dense Perceptioncited by 33
  6. 2023LP-OVOD: Open-Vocabulary Object Detection by Linear Probingcited by 29
  7. 2023Expanding Scene Graph Boundaries: Fully Open-vocabulary Scene Graph Generation via Visual-Concept Alignment and Retentioncited by 29
  8. 2024Generative Region-Language Pretraining for Open-Ended Object Detectioncited by 29
  9. 2024OV-DINO: Unified Open-Vocabulary Detection with Language-Aware Selective Fusioncited by 28
  10. 2024GET: Unlocking the Multi-modal Potential of CLIP for Generalized Category Discoverycited by 27
  11. 2024InstaGen: Enhancing Object Detection by Training on Synthetic Datasetcited by 26
  12. 2023Taming Self-Training for Open-Vocabulary Object Detectioncited by 25
  13. 2023Object2Scene: Putting Objects in Context for Open-Vocabulary 3D Detectioncited by 24
  14. 2024Learning Background Prompts to Discover Implicit Knowledge for Open Vocabulary Object Detectioncited by 24
  15. 2023Toward Open Vocabulary Aerial Object Detection with CLIP-Activated Student-Teacher Learningcited by 22
  16. 2024OV-DQUO: Open-Vocabulary DETR with Denoising Text Query Training and Open-World Unknown Objects Supervisioncited by 22
  17. 2023Zero-shot Referring Expression Comprehension via Structural Similarity Between Images and Captionscited by 21
  18. 2023CLIM: Contrastive Language-Image Mosaic for Region Representationcited by 21
  19. 2025Referring to Any Personcited by 21
  20. 2025LPOSS: Label Propagation Over Patches and Pixels for Open-vocabulary Semantic Segmentationcited by 21
  21. 2024Real-time Transformer-based Open-Vocabulary Detection with Efficient Fusion Headcited by 20
  22. 2024Retrieval-Augmented Open-Vocabulary Object Detectioncited by 20
  23. 2024LaMI-DETR: Open-Vocabulary Detection with Language Model Instructioncited by 19
  24. 2024Scene Graph Generation with Role-Playing Large Language Modelscited by 16
  25. 2023Simple Image-level Classification Improves Open-vocabulary Object Detectioncited by 15
  26. 2026Vision Transformers Need More Than Registerscited by 15
  27. 2023Detect Everything with Few Examplescited by 14
  28. 2024VisionGPT: LLM-Assisted Real-Time Anomaly Detection for Safe Visual Navigationcited by 14
  29. 2025Object Detection with Multimodal Large Vision-Language Models: An In-depth Reviewcited by 14
  30. 2024Open-Set Recognition in the Age of Vision-Language Modelscited by 13
  31. 2024Frozen-DETR: Enhancing DETR with Image Understanding from Frozen Foundation Modelscited by 13
  32. 2025Refining CLIP's Spatial Awareness: A Visual-Centric Perspectivecited by 13
  33. 2024Hyperbolic Learning with Synthetic Captions for Open-World Detectioncited by 12
  34. 2025Patch Matters: Training-free Fine-grained Image Caption Enhancement via Local Perceptioncited by 12
  35. 2024SHiNe: Semantic Hierarchy Nexus for Open-vocabulary Object Detectioncited by 11
  36. 2024DiPEx: Dispersing Prompt Expansion for Class-Agnostic Object Detectioncited by 11
  37. 2024MarvelOVD: Marrying Object Recognition and Vision-Language Models for Robust Open-Vocabulary Object Detectioncited by 11
  38. 2023Boosting Segment Anything Model Towards Open-Vocabulary Learningcited by 10

50 most cited of 119 All 119 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Sampling Bag of Views for Open-Vocabulary Object Detection
  2. 2022X2-VLM: All-In-One Pre-trained Model For Vision-Language Tasks
  3. 2022F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Models
  4. 2023Three ways to improve feature alignment for open vocabulary detection
  5. 2023COPA: Efficient Vision-Language Pre-training Through Collaborative Object- and Patch-Text Alignment
  6. 2023DetCLIPv2: Scalable Open-Vocabulary Object Detection Pre-training via Word-Region Alignment
  7. 2023EdaDet: Open-Vocabulary Object Detection Using Early Dense Alignment
  8. 2024OLIVE: Object Level In-Context Visual Embeddings