mlsanityData to 14 Oct 2026

Region-Aware Pretraining for Open-Vocabulary Object Detection with Vision Transformers

arXiv 2305.07011 ↗First posted 11 May 2023cs.CV cs.AI cs.CLcited by 91

Not on the map. Why?

Abstract (arXiv)

We present Region-aware Open-vocabulary Vision Transformers (RO-ViT) - a contrastive image-text pretraining recipe to bridge the gap between image-level pretraining and open-vocabulary object detection. At the pretraining phase, we propose to randomly crop and resize regions of positional embeddings instead of using the whole image positional embeddings. This better matches the use of positional embeddings at region-level in the detection finetuning phase. In addition, we replace the common softmax cross entropy loss in contrastive learning with focal loss to better learn the informative yet difficult examples. Finally, we leverage recent advances in novel object proposals to improve open-vocabulary detection finetuning. We evaluate our full model on the LVIS and COCO open-vocabulary detection benchmarks and zero-shot transfer. RO-ViT achieves a state-of-the-art 34.1 APr on LVIS, surpassing the best existing approach by +7.8 points in addition to competitive zero-shot transfer detection. Surprisingly, RO-ViT improves the image-level representation as well and achieves the state of the art on 9 out of 12 metrics on COCO and Flickr image-text retrieval benchmarks, outperforming competitive approaches with larger models.

Key facts. Cited by 91 papers in the map (data through 14 Oct 2026). Builds on 61 of its 70 references that are in the map. First posted on arXiv 11 May 2023.

Builds on

61 of 70 references.

References · 61

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  7. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  8. 2021ViViT: A Video Vision Transformercited by 1,620
  9. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  10. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  11. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  12. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
38 more
  1. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,433
  2. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  3. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  4. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,217
  5. 2021Grounded Language-Image Pre-trainingcited by 1,156
  6. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 906
  7. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  8. 2017Focal Loss for Dense Object Detectioncited by 752
  9. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 724
  10. 2022Language-driven Semantic Segmentationcited by 717
  11. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
  12. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 647
  13. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 627
  14. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 616
  15. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 614
  16. 2020Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentationcited by 596
  17. 2021Extract Free Dense Labels from CLIPcited by 583
  18. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
  19. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 473
  20. 2020Open-Vocabulary Object Detection Using Captionscited by 427
  21. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 339
  22. 2022Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Modelcited by 317
  23. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 305
  24. 2022Open-Vocabulary DETR with Conditional Matchingcited by 220
  25. 2018Zero-Shot Object Detectioncited by 189
  26. 2021Combined Scaling for Zero-shot Transfer Learningcited by 182
  27. 2015Learning Visual Features from Large Weakly Supervised Datacited by 165
  28. 2021Efficient Visual Pretraining with Contrastive Detectioncited by 150
  29. 2022F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Modelscited by 150
  30. 2020Seesaw Loss for Long-Tailed Instance Segmentationcited by 146
  31. 2022Bridging the Gap between Object and Image-level Representations for Open-Vocabulary Detectioncited by 146
  32. 2013Zero-Shot Learning by Convex Combination of Semantic Embeddingscited by 135
  33. 2022PromptDet: Towards Open-vocabulary Detection using Uncurated Imagescited by 133
  34. 2020Learning Visual Representations with Caption Annotationscited by 123
  35. 2021Aligning Pretraining for Detection via Object-Level Contrastive Learningcited by 110
  36. 2021Learning Open-World Object Proposals without Learning to Classifycited by 109
  37. 2022SegViT: Semantic Segmentation with Plain Vision Transformerscited by 106
  38. 2022Exploiting Unlabeled Data with Vision and Language Models for Object Detectioncited by 97

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 17; 2024: 30; 2025: 27; 2026: 17.

91 papers.

Sort:

By area:

  1. 2024Long-CLIP: Unlocking the Long-Text Capability of CLIPcited by 300
  2. 2023AM-RADIO: Agglomerative Vision Foundation Model -- Reduce All Domains Into Onecited by 169
  3. 2023Towards Open Vocabulary Learning: A Surveycited by 166
  4. 2023CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Predictioncited by 118
  5. 2024RADIOv2.5: Improved Baselines for Agglomerative Vision Foundation Modelscited by 89
  6. 2025FG-CLIP: Fine-Grained Visual and Textual Alignmentcited by 81
  7. 2023A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Futurecited by 61
  8. 2023Open-Vocabulary Video Anomaly Detectioncited by 57
  9. 2024MaskClustering: View Consensus based Mask Graph Clustering for Open-Vocabulary 3D Instance Segmentationcited by 56
  10. 2024VideoCLIP-XL: Advancing Long Description Understanding for Video CLIP Modelscited by 41
  11. 2024ViTamin: Designing Scalable Vision Models in the Vision-Language Eracited by 39
  12. 2024Diffusion Feedback Helps CLIP See Bettercited by 38
38 more
  1. 2025DeCLIP: Decoupled Learning for Open-Vocabulary Dense Perceptioncited by 33
  2. 2023Contrastive Feature Masking Open-Vocabulary Vision Transformercited by 31
  3. 2024Generative Region-Language Pretraining for Open-Ended Object Detectioncited by 29
  4. 2025Exploring Unbiased Deepfake Detection via Token-Level Shuffling and Mixingcited by 27
  5. 2024InstaGen: Enhancing Object Detection by Training on Synthetic Datasetcited by 26
  6. 2024AIDE: An Automatic Data Engine for Object Detection in Autonomous Drivingcited by 26
  7. 2023Taming Self-Training for Open-Vocabulary Object Detectioncited by 25
  8. 2024Learning Background Prompts to Discover Implicit Knowledge for Open Vocabulary Object Detectioncited by 24
  9. 2023Toward Open Vocabulary Aerial Object Detection with CLIP-Activated Student-Teacher Learningcited by 22
  10. 2024OV-DQUO: Open-Vocabulary DETR with Denoising Text Query Training and Open-World Unknown Objects Supervisioncited by 22
  11. 2024LaMI-DETR: Open-Vocabulary Detection with Language Model Instructioncited by 19
  12. 2024Adapting Pre-Trained Vision Models for Novel Instance Detection and Segmentationcited by 17
  13. 2026Vision Transformers Need More Than Registerscited by 15
  14. 2023Detect Everything with Few Examplescited by 14
  15. 2024USE: Universal Segment Embeddings for Open-Vocabulary Image Segmentationcited by 14
  16. 2025Object Detection with Multimodal Large Vision-Language Models: An In-depth Reviewcited by 14
  17. 2024Open World Object Detection: A Surveycited by 13
  18. 2025Anomize: Better Open Vocabulary Video Anomaly Detectioncited by 13
  19. 2025Refining CLIP's Spatial Awareness: A Visual-Centric Perspectivecited by 13
  20. 2024UNIT: Unifying Image and Text Recognition in One Vision Encodercited by 12
  21. 2025Vision Transformers with Self-Distilled Registerscited by 12
  22. 2024A Review of Transformer-Based Models for Computer Vision Tasks: Capturing Global Context and Spatial Relationshipscited by 11
  23. 2025Vision-Language Model for Object Detection and Segmentation: A Review and Evaluationcited by 10
  24. 2025Generalized Decoupled Learning for Enhancing Open-Vocabulary Dense Perceptioncited by 10
  25. 2024Advancing Molecular Graph-Text Pre-training via Fine-grained Alignmentcited by 9
  26. 2024Unbiased Region-Language Alignment for Open-Vocabulary Dense Predictioncited by 9
  27. 2023Region-centric Image-Language Pretraining for Open-Vocabulary Detectioncited by 8
  28. 2024Exploiting VLM Localizability and Semantics for Open Vocabulary Action Detectioncited by 8
  29. 2025Grad-ECLIP: Gradient-based Visual and Textual Explanations for CLIPcited by 8
  30. 2025VAGU & GtS: LLM-Based Benchmark and Framework for Joint Video Anomaly Grounding and Understandingcited by 8
  31. 2025Open-Vocabulary HOI Detection with Interaction-aware Prompt and Concept Calibrationcited by 8
  32. 2023Rethinking Evaluation Metrics of Open-Vocabulary Segmentaioncited by 7
  33. 2023Object-Centric Open-Vocabulary Image-Retrieval with Aggregated Featurescited by 5
  34. 2023MarineDet: Towards Open-Marine Object Detectioncited by 5
  35. 2023Language-conditioned Detection Transformercited by 5
  36. 2024SIA-OVD: Shape-Invariant Adapter for Bridging the Image-Region Gap in Open-Vocabulary Detectioncited by 5
  37. 2025C3-OWD: A Curriculum Cross-modal Contrastive Learning Framework for Open-World Detectioncited by 5
  38. 2023Unsupervised Camouflaged Object Segmentation as Domain Adaptationcited by 4

50 most cited of 91 All 91 (JSON)