mlsanityData to 7 Oct 2026

Open-Vocabulary DETR with Conditional Matching

arXiv 2203.11876 ↗First posted 22 Mar 2022cs.CV cs.AIcited by 220

Not on the map. Why?

Cited by 44 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Open-vocabulary object detection, which is concerned with the problem of detecting novel objects guided by natural language, has gained increasing attention from the community. Ideally, we would like to extend an open-vocabulary detector such that it can produce bounding box predictions based on user inputs in form of either natural language or exemplar image. This offers great flexibility and user experience for human-computer interaction. To this end, we propose a novel open-vocabulary detector based on DETR -- hence the name OV-DETR -- which, once trained, can detect any object given its class name or an exemplar image. The biggest challenge of turning DETR into an open-vocabulary detector is that it is impossible to calculate the classification cost matrix of novel classes without access to their labeled images. To overcome this challenge, we formulate the learning objective as a binary matching one between input queries (class name or exemplar image) and the corresponding objects, which learns useful correspondence to generalize to unseen queries during testing. For training, we choose to condition the Transformer decoder on the input embeddings obtained from a pre-trained vision-language model like CLIP, in order to enable matching for both text and image queries. With extensive experiments on LVIS and COCO datasets, we demonstrate that our OV-DETR -- the first end-to-end Transformer-based open-vocabulary detector -- achieves non-trivial improvements over current state of the arts.

Key facts. Cited by 220 papers in the map (data through 7 Oct 2026). Builds on 28 of its 42 references that are in the map. First posted on arXiv 22 Mar 2022.

Builds on

28 of 42 references.

References · 28

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  5. 2020End-to-End Object Detection with Transformerscited by 7,384
  6. 2017Mask R-CNNcited by 7,259
  7. 2022Generative Adversarial Networkscited by 5,603
  8. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  9. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  10. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  11. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  12. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
16 more
  1. 2019Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regressioncited by 1,361
  2. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  3. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  4. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  5. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 613
  6. 2020Open-Vocabulary Object Detection Using Captionscited by 427
  7. 2021Conditional DETR for Fast Training Convergencecited by 422
  8. 2020Sparse Sinkhorn Attentioncited by 250
  9. 2018Zero-Shot Object Detectioncited by 189
  10. 2021Fast Convergence of DETR with Spatially Modulated Co-Attentioncited by 176
  11. 2017Person Search with Natural Language Descriptioncited by 149
  12. 2021Learning Open-World Object Proposals without Learning to Classifycited by 109
  13. 2019Dont Even Look Once: Synthesizing Features for Zero-Shot Detectioncited by 58
  14. 2017Query-guided Regression Network with Context Policy for Phrase Groundingcited by 34
  15. 2020GTNet: Generative Transfer Network for Zero-Shot Object Detectioncited by 25
  16. 2020An Improved Attention for Visual Question Answeringcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 10; 2023: 68; 2024: 70; 2025: 43; 2026: 29.

220 papers.

Sort:

By area:

  1. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  2. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  3. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  4. 2024YOLO-World: Real-Time Open-Vocabulary Object Detectioncited by 498
  5. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Modelscited by 371
  6. 2023Recognize Anything: A Strong Image Tagging Modelcited by 305
  7. 2023Self-regulating Prompts: Foundational Model Adaptation without Forgettingcited by 272
  8. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  9. 2022DetCLIP: Dictionary-Enriched Visual-Concept Paralleled Pre-training for Open-world Detectioncited by 176
  10. 2022PLA: Language-Driven Open-Vocabulary 3D Scene Understandingcited by 176
  11. 2023Towards Open Vocabulary Learning: A Surveycited by 165
  12. 2022Unified Vision and Language Prompt Learningcited by 145
38 more
  1. 2023CORA: Adapting CLIP for Open-Vocabulary Detection with Region Prompting and Anchor Pre-Matchingcited by 133
  2. 2022Open-Vocabulary Universal Image Segmentation with MaskCLIPcited by 131
  3. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
  4. 2023Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidancecited by 123
  5. 2023Aligning Bag of Regions for Open-Vocabulary Object Detectioncited by 119
  6. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  7. 2023CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Predictioncited by 114
  8. 2023Iterative Prompt Learning for Unsupervised Backlit Image Enhancementcited by 107
  9. 2023Contextual Object Detection with Multimodal Large Language Modelscited by 100
  10. 2024T-Rex2: Towards Generic Object Detection via Text-Visual Prompt Synergycited by 94
  11. 2023Region-Aware Pretraining for Open-Vocabulary Object Detection with Vision Transformerscited by 91
  12. 2023DetCLIPv2: Scalable Open-Vocabulary Object Detection Pre-training via Word-Region Alignmentcited by 90
  13. 2023CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIPcited by 89
  14. 2023Fine-Grained Visual Promptingcited by 81
  15. 2024Open-Vocabulary SAM: Segment and Recognize Twenty-thousand Classes Interactivelycited by 74
  16. 2024OMG-Seg: Is One Model Good Enough For All Segmentation?cited by 73
  17. 2023Object-Aware Distillation Pyramid for Open-Vocabulary Object Detectioncited by 63
  18. 2023Detecting Everything in the Open World: Towards Universal Object Detectioncited by 62
  19. 2023HOICLIP: Efficient Knowledge Transfer for HOI Detection with Vision-Language Modelscited by 60
  20. 2023A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Futurecited by 59
  21. 2023Multi-Modal Classifiers for Open-Vocabulary Object Detectioncited by 53
  22. 2023Visual In-Context Promptingcited by 52
  23. 2024DetCLIPv3: Towards Versatile Generative Open-vocabulary Object Detectioncited by 48
  24. 2023Multi-modal Queried Object Detection in the Wildcited by 47
  25. 2026Ultralytics YOLO26: Unified Real-Time End-to-End Vision Modelscited by 47
  26. 2023CoDet: Co-Occurrence Guided Region-Word Alignment for Open-Vocabulary Object Detectioncited by 46
  27. 2023Towards Open-Vocabulary Video Instance Segmentationcited by 45
  28. 2024DetToolChain: A New Prompting Paradigm to Unleash Detection Ability of MLLMcited by 45
  29. 2023EdaDet: Open-Vocabulary Object Detection Using Early Dense Alignmentcited by 44
  30. 2024vid-TLDR: Training Free Token merging for Light-weight Video Transformercited by 41
  31. 2023Described Object Detection: Liberating Object Detection with Flexible Expressionscited by 40
  32. 2023Point, Segment and Count: A Generalized Framework for Object Countingcited by 39
  33. 2024ViTamin: Designing Scalable Vision Models in the Vision-Language Eracited by 39
  34. 2024Harnessing Vision Foundation Models for High-Performance, Training-Free Open Vocabulary Segmentationcited by 38
  35. 2025YOLOE: Real-Time Seeing Anythingcited by 36
  36. 2023MosaicFusion: Diffusion Models as Data Augmenters for Large Vocabulary Instance Segmentationcited by 35
  37. 2022Open-Vocabulary Multi-Label Classification via Multi-Modal Knowledge Transfercited by 33
  38. 2025DeCLIP: Decoupled Learning for Open-Vocabulary Dense Perceptioncited by 33

50 most cited of 220 All 220 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024OVLW-DETR: Open-Vocabulary Light-Weighted Detection Transformer
  2. 2023What Makes Good Open-Vocabulary Detector: A Disassembling Perspective
  3. 2021Open Vocabulary Object Detection with Pseudo Bounding-Box Labels
  4. 2023Open-Vocabulary Object Detection using Pseudo Caption Labels
  5. 2022Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Model
  6. 2024From Open Vocabulary to Open World: Teaching Vision Language Models to Detect Novel Objects
  7. 2024SIA-OVD: Shape-Invariant Adapter for Bridging the Image-Region Gap in Open-Vocabulary Detection
  8. 2022Open Vocabulary Object Detection with Proposal Mining and Prediction Equalization