mlsanityData to 7 Oct 2026

Scaling Open-Vocabulary Object Detection

arXiv 2306.09683 ↗First posted 16 Jun 2023cs.CVcited by 335

On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection

Abstract (arXiv)

Open-vocabulary object detection has benefited greatly from pretrained vision-language models, but is still limited by the amount of available detection training data. While detection training data can be expanded by using Web image-text pairs as weak supervision, this has not been done at scales comparable to image-level pretraining. Here, we scale up detection data with self-training, which uses an existing detector to generate pseudo-box annotations on image-text pairs. Major challenges in scaling self-training are the choice of label space, pseudo-annotation filtering, and training efficiency. We present the OWLv2 model and OWL-ST self-training recipe, which address these challenges. OWLv2 surpasses the performance of previous state-of-the-art open-vocabulary detectors already at comparable training scales (~10M examples). However, with OWL-ST, we can scale to over 1B examples, yielding further large improvement: With an L/14 architecture, OWL-ST improves AP on LVIS rare classes, for which the model has seen no human box annotations, from 31.2% to 44.6% (43% relative improvement). OWL-ST unlocks Web-scale training for open-world localization, similar to what has been seen for image classification and language modelling.

Key facts. Cited by 335 papers in the map (data through 7 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 40 of its 49 references that are in the map. First posted on arXiv 16 Jun 2023.

Builds on

40 of 49 references.

References · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2023Segment Anythingcited by 8,158
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  7. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  9. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  10. 2021Grounded Language-Image Pre-trainingcited by 1,147
  11. 2018Model Cards for Model Reportingcited by 1,068
  12. 2022Token Merging: Your ViT But Fastercited by 995
28 more
  1. 2021Scaling Vision Transformerscited by 993
  2. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  3. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  4. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  5. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  6. 2021Robust fine-tuning of zero-shot modelscited by 785
  7. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  8. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 723
  9. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  10. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 613
  11. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 612
  12. 2020Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentationcited by 595
  13. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  14. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 542
  15. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 360
  16. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 359
  17. 2020Rethinking Pre-training and Self-trainingcited by 352
  18. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 303
  19. 2020A Simple Semi-Supervised Learning Framework for Object Detectioncited by 282
  20. 2022GLIPv2: Unifying Localization and Vision-Language Understandingcited by 275
  21. 2021Combined Scaling for Zero-shot Transfer Learningcited by 182
  22. 2022DetCLIP: Dictionary-Enriched Visual-Concept Paralleled Pre-training for Open-world Detectioncited by 176
  23. 2022ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Modelscited by 141
  24. 2021Efficient DETR: Improving End-to-End Object Detector with Dense Priorcited by 128
  25. 2023DetCLIPv2: Scalable Open-Vocabulary Object Detection Pre-training via Word-Region Alignmentcited by 90
  26. 2021Evaluating Large-Vocabulary Object Detectors: The Devil is in the Detailscited by 57
  27. 2021SCENIC: A JAX Library for Computer Vision Research and Beyondcited by 56
  28. 2023Three ways to improve feature alignment for open vocabulary detectioncited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 22; 2024: 91; 2025: 129; 2026: 93.

335 papers.

Sort:

By area:

  1. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
  2. 2024YOLO-World: Real-Time Open-Vocabulary Object Detectioncited by 498
  3. 2024Robotic Control via Embodied Chain-of-Thought Reasoningcited by 340
  4. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 315
  5. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decodingcited by 171
  6. 2023Towards Open Vocabulary Learning: A Surveycited by 165
  7. 2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planningcited by 161
  8. 2025HAMSTER: Hierarchical Action Models For Open-World Robot Manipulationcited by 125
  9. 2024Grounding DINO 1.5: Advance the "Edge" of Open-Set Object Detectioncited by 122
  10. 2023Self-correcting LLM-controlled Diffusion Modelscited by 112
  11. 2024Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Modelscited by 90
  12. 2024DINO-X: A Unified Vision Model for Open-World Object Detection and Understandingcited by 85
38 more
  1. 2025Describe Anything: Detailed Localized Image and Video Captioningcited by 84
  2. 2024Benchmarking and Improving Detail Image Captioncited by 81
  3. 2025FG-CLIP: Fine-Grained Visual and Textual Alignmentcited by 78
  4. 2025BridgeVLA: Input-Output Alignment for Efficient 3D Manipulation Learning with Vision-Language Modelscited by 77
  5. 2024No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performancecited by 75
  6. 2024DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal Perceptioncited by 72
  7. 2024HM3D-OVON: A Dataset and Benchmark for Open-Vocabulary Object Goal Navigationcited by 71
  8. 2025ConceptMaster: Multi-Concept Video Customization on Diffusion Transformer Models Without Test-Time Tuningcited by 67
  9. 2023SILC: Improving Vision Language Pretraining with Self-Distillationcited by 66
  10. 2025Detect Anything via Next Point Predictioncited by 66
  11. 2024PoliFormer: Scaling On-Policy RL with Transformers Results in Masterful Navigatorscited by 55
  12. 2024Visual Haystacks: A Vision-Centric Needle-In-A-Haystack Benchmarkcited by 53
  13. 2025LLMDet: Learning Strong Open-Vocabulary Object Detectors under the Supervision of Large Language Modelscited by 53
  14. 2024LLaRA: Supercharging Robot Learning Data for Vision-Language Policycited by 51
  15. 2025Interleave-VLA: Enhancing Robot Manipulation with Interleaved Image-Text Instructionscited by 50
  16. 2024DetCLIPv3: Towards Versatile Generative Open-vocabulary Object Detectioncited by 48
  17. 2024Talking to DINO: Bridging Self-Supervised Vision Backbones with Language for Open-Vocabulary Segmentationcited by 42
  18. 2023TokenCompose: Text-to-Image Diffusion with Token-level Supervisioncited by 40
  19. 2026Image Generators are Generalist Vision Learnerscited by 40
  20. 2024Towards Generalizable Vision-Language Robotic Manipulation: A Benchmark and LLM-guided 3D Policycited by 37
  21. 2024MoAI: Mixture of All Intelligence for Large Language and Vision Modelscited by 36
  22. 2024Enhancing Video-LLM Reasoning via Agent-of-Thoughts Distillationcited by 36
  23. 2024V-IRL: Grounding Virtual Intelligence in Real Lifecited by 35
  24. 2024LocCa: Visual Pretraining with Location-aware Captionerscited by 35
  25. 2024Identity Decoupling for Multi-Subject Personalization of Text-to-Image Modelscited by 33
  26. 2024Contrastive Localized Language-Image Pre-Trainingcited by 33
  27. 2023Revisiting Few-Shot Object Detection with Vision-Language Modelscited by 32
  28. 2024DynaMem: Online Dynamic Spatio-Semantic Memory for Open World Mobile Manipulationcited by 32
  29. 2023Tokenize Anything via Promptingcited by 31
  30. 2025Exploring the Limits of Vision-Language-Action Manipulations in Cross-task Generalizationcited by 31
  31. 2023Segment and Caption Anythingcited by 30
  32. 2023The devil is in the fine-grained details: Evaluating open-vocabulary object detectors for fine-grained understandingcited by 29
  33. 2024Enhancing Semantic Fidelity in Text-to-Image Synthesis: Attention Regulation in Diffusion Modelscited by 27
  34. 2024AIDE: An Automatic Data Engine for Object Detection in Autonomous Drivingcited by 26
  35. 2024TANGO: Training-free Embodied AI Agents for Open-world Taskscited by 26
  36. 2024VILA2: VILA Augmented VILAcited by 25
  37. 2024FreeTuner: Any Subject in Any Style with Training-free Diffusioncited by 24
  38. 2024MLLM-as-a-Judge for Image Safety without Human Labelingcited by 23

50 most cited of 335 All 335 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Taming Self-Training for Open-Vocabulary Object Detection
  2. 2024MarvelOVD: Marrying Object Recognition and Vision-Language Models for Robust Open-Vocabulary Object Detection
  3. 2021Open Vocabulary Object Detection with Pseudo Bounding-Box Labels
  4. 2026Exploring Hierarchical Consistency and Unbiased Objectness for Open-Vocabulary Object Detection
  5. 2026NoOVD: Novel Category Discovery and Embedding for Open-Vocabulary Object Detection
  6. 2023EdaDet: Open-Vocabulary Object Detection Using Early Dense Alignment
  7. 2022F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Models
  8. 2023Open-Vocabulary Object Detection using Pseudo Caption Labels