mlsanityData to 7 Oct 2026

Simple Open-Vocabulary Object Detection with Vision Transformers

arXiv 2205.06230 ↗First posted 12 May 2022cs.CVcited by 303

Not on the map. Why?

Cited by 61 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Combining simple architectures with large-scale pre-training has led to massive improvements in image classification. For object detection, pre-training and scaling approaches are less well established, especially in the long-tailed and open-vocabulary setting, where training data is relatively scarce. In this paper, we propose a strong recipe for transferring image-text models to open-vocabulary object detection. We use a standard Vision Transformer architecture with minimal modifications, contrastive image-text pre-training, and end-to-end detection fine-tuning. Our analysis of the scaling properties of this setup shows that increasing image-level pre-training and model size yield consistent improvements on the downstream detection task. We provide the adaptation strategies and regularizations needed to attain very strong performance on zero-shot text-conditioned and one-shot image-conditioned object detection. Code and models are available on GitHub.

Key facts. Cited by 303 papers in the map (data through 7 Oct 2026). Builds on 43 of its 49 references that are in the map. First posted on arXiv 12 May 2022.

Builds on

43 of 49 references.

References · 43

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  6. 2020End-to-End Object Detection with Transformerscited by 7,384
  7. 2017Mask R-CNNcited by 7,259
  8. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  10. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  11. 2015SSD: Single Shot MultiBox Detectorcited by 3,125
  12. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
31 more
  1. 2021ViViT: A Video Vision Transformercited by 1,610
  2. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  3. 2021Grounded Language-Image Pre-trainingcited by 1,147
  4. 2021Scaling Vision Transformerscited by 993
  5. 2016Deep Networks with Stochastic Depthcited by 973
  6. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  7. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  8. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  9. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 613
  10. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 612
  11. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
  12. 2020Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentationcited by 595
  13. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  14. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  15. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 542
  16. 2017Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Uglycited by 536
  17. 2020Open-Vocabulary Object Detection Using Captionscited by 427
  18. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  19. 2013Zero-Shot Learning Through Cross-Modal Transfercited by 237
  20. 2018Zero-Shot Object Detectioncited by 189
  21. 2021Combined Scaling for Zero-shot Transfer Learningcited by 182
  22. 2021You Only Look at One Sequence: Rethinking Transformer in Vision through Object Detectioncited by 170
  23. 2021Revisiting ResNets: Improved Training and Scaling Strategiescited by 158
  24. 2021Probabilistic two-stage detectioncited by 130
  25. 2018Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networkscited by 129
  26. 2021Efficient DETR: Improving End-to-End Object Detector with Dense Priorcited by 128
  27. 2019One-Shot Object Detection with Co-Attention and Co-Excitationcited by 67
  28. 2021SCENIC: A JAX Library for Computer Vision Research and Beyondcited by 56
  29. 2021ViDT: An Efficient and Effective Fully Transformer-based Object Detectorcited by 45
  30. 2018One-Shot Instance Segmentationcited by 43
  31. 2020OS2D: One-Stage One-Shot Object Detection by Matching Anchor Featurescited by 35

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 24; 2023: 108; 2024: 90; 2025: 44; 2026: 37.

303 papers.

Sort:

By area:

  1. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  2. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  3. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 946
  4. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  5. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  6. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  7. 2023LERF: Language Embedded Radiance Fieldscited by 538
  8. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  9. 2024YOLO-World: Real-Time Open-Vocabulary Object Detectioncited by 498
  10. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 459
  11. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  12. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Modelscited by 371
38 more
  1. 2023Scaling Open-Vocabulary Object Detectioncited by 335
  2. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 315
  3. 2023Training-Free Layout Control with Cross-Attention Guidancecited by 312
  4. 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulationcited by 305
  5. 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 298
  6. 2022Generalized Decoding for Pixel, Image, and Languagecited by 292
  7. 2023Aligning Text-to-Image Diffusion Models with Reward Backpropagationcited by 284
  8. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  9. 2022OpenOOD: Benchmarking Generalized Out-of-Distribution Detectioncited by 267
  10. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 254
  11. 2023A Cookbook of Self-Supervised Learningcited by 248
  12. 2023LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Modelscited by 242
  13. 2023Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolutioncited by 226
  14. 2023TidyBot: Personalized Robot Assistance with Large Language Modelscited by 219
  15. 2023RSPrompter: Learning to Prompt for Remote Sensing Instance Segmentation based on Visual Foundation Modelcited by 180
  16. 2023Scaling Robot Learning with Semantically Imagined Experiencecited by 175
  17. 2023RoCo: Dialectic Multi-Robot Collaboration with Large Language Modelscited by 171
  18. 2023RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunkingcited by 167
  19. 2023Towards Open Vocabulary Learning: A Surveycited by 165
  20. 2023Physically Grounded Vision-Language Models for Robotic Manipulationcited by 163
  21. 2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planningcited by 161
  22. 2023Open-World Object Manipulation using Pre-trained Vision-Language Modelscited by 155
  23. 2022FlexiViT: One Model for All Patch Sizescited by 151
  24. 2022F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Modelscited by 149
  25. 2024Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoningcited by 136
  26. 2024Grounding DINO 1.5: Advance the "Edge" of Open-Set Object Detectioncited by 122
  27. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  28. 2023CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Predictioncited by 114
  29. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  30. 2024SG-Nav: Online 3D Scene Graph Prompting for LLM-based Zero-shot Object Navigationcited by 106
  31. 2024ZoomEye: Enhancing Multimodal LLMs with Human-Like Zooming Capabilities through Tree-Based Image Explorationcited by 99
  32. 2022Benchmarking Spatial Relationships in Text-to-Image Generationcited by 96
  33. 2022Learning Object-Language Alignments for Open-Vocabulary Object Detectioncited by 95
  34. 2023Vision-Language Models in Remote Sensing: Current Progress and Future Trendscited by 93
  35. 2023Region-Aware Pretraining for Open-Vocabulary Object Detection with Vision Transformerscited by 91
  36. 2023Language Embedded Radiance Fields for Zero-Shot Task-Oriented Graspingcited by 91
  37. 2024FMGS: Foundation Model Embedded 3D Gaussian Splatting for Holistic 3D Scene Understandingcited by 86
  38. 2024DINO-X: A Unified Vision Model for Open-World Object Detection and Understandingcited by 85

50 most cited of 303 All 303 (JSON)