mlsanityData to 7 Oct 2026

LaMI-DETR: Open-Vocabulary Detection with Language Model Instruction

arXiv 2407.11335 ↗First posted 16 Jul 2024cs.CVcited by 19

On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection

Abstract (arXiv)

Existing methods enhance open-vocabulary object detection by leveraging the robust open-vocabulary recognition capabilities of Vision-Language Models (VLMs), such as CLIP.However, two main challenges emerge:(1) A deficiency in concept representation, where the category names in CLIP's text space lack textual and visual knowledge.(2) An overfitting tendency towards base categories, with the open vocabulary knowledge biased towards base categories during the transfer from VLMs to detectors.To address these challenges, we propose the Language Model Instruction (LaMI) strategy, which leverages the relationships between visual concepts and applies them within a simple yet effective DETR-like detector, termed LaMI-DETR.LaMI utilizes GPT to construct visual concepts and employs T5 to investigate visual similarities across categories.These inter-category relationships refine concept representation and avoid overfitting to base categories.Comprehensive experiments validate our approach's superior performance over existing methods in the same rigorous setting without reliance on external training resources.LaMI-DETR achieves a rare box AP of 43.4 on OV-LVIS, surpassing the previous best by 7.8 rare box AP.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 39 of its 47 references that are in the map. First posted on arXiv 16 Jul 2024.

Builds on

39 of 47 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023EdaDet: Open-Vocabulary Object Detection Using Early Dense Alignment3 Sep
  2. 2023Scaling Open-Vocabulary Object Detection16 Jun
  3. 2023Aligning Bag of Regions for Open-Vocabulary Object Detection27 Feb
  4. 2022Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Model28 Mar

Other references · 35

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022A ConvNet for the 2020scited by 3,545
  4. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  5. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,306
  6. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  7. 2021Grounded Language-Image Pre-trainingcited by 1,147
  8. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  9. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 613
  10. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 612
  11. 2020Open-Vocabulary Object Detection Using Captionscited by 427
  12. 2022Visual Classification via Description from Large Language Modelscited by 331
23 more
  1. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 329
  2. 2022Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classificationcited by 310
  3. 2022What does a platypus look like? Generating customized prompts for zero-shot image classificationcited by 263
  4. 2022Open-Vocabulary DETR with Conditional Matchingcited by 220
  5. 2018Zero-Shot Object Detectioncited by 189
  6. 2022DetCLIP: Dictionary-Enriched Visual-Concept Paralleled Pre-training for Open-world Detectioncited by 176
  7. 2022Bridging the Gap between Object and Image-level Representations for Open-Vocabulary Detectioncited by 145
  8. 2022PromptDet: Towards Open-vocabulary Detection using Uncurated Imagescited by 133
  9. 2023CORA: Adapting CLIP for Open-Vocabulary Detection with Region Prompting and Anchor Pre-Matchingcited by 133
  10. 2021Probabilistic two-stage detectioncited by 130
  11. 2022Exploiting Unlabeled Data with Vision and Language Models for Object Detectioncited by 97
  12. 2023Region-Aware Pretraining for Open-Vocabulary Object Detection with Vision Transformerscited by 91
  13. 2023DetCLIPv2: Scalable Open-Vocabulary Object Detection Pre-training via Word-Region Alignmentcited by 90
  14. 2023Object-Aware Distillation Pyramid for Open-Vocabulary Object Detectioncited by 63
  15. 2019Dont Even Look Once: Synthesizing Features for Zero-Shot Detectioncited by 58
  16. 2021Evaluating Large-Vocabulary Object Detectors: The Devil is in the Detailscited by 57
  17. 2023CoDet: Co-Occurrence Guided Region-Word Alignment for Open-Vocabulary Object Detectioncited by 46
  18. 2024LW-DETR: A Transformer Replacement to YOLO for Real-Time Detectioncited by 36
  19. 2024MS-DETR: Efficient DETR Training with Mixed Supervisioncited by 34
  20. 2018Zero-Shot Object Detection by Hybrid Region Embeddingcited by 32
  21. 2023Contrastive Feature Masking Open-Vocabulary Vision Transformercited by 31
  22. 2020GTNet: Generative Transfer Network for Zero-Shot Object Detectioncited by 25
  23. 2022Learning to Detect and Segment for Open Vocabulary Object Detectioncited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 12; 2026: 6.

19 papers.

Sort:

By area:

  1. 2025OFFSET: Segmentation-based Focus Shift Revision for Composed Image Retrievalcited by 40
  2. 2025HUD: Hierarchical Uncertainty-Aware Disambiguation Network for Composed Video Retrievalcited by 32
  3. 2025WeDetect: Fast Open-Vocabulary Object Detection as Retrievalcited by 11
  4. 2025Vision-Language Model for Object Detection and Segmentation: A Review and Evaluationcited by 10
  5. 2024Enhancing Descriptive Captions with Visual Attributes for Multimodal Perceptioncited by 9
  6. 2025Adaptive Event Stream Slicing for Open-Vocabulary Event-Based Object Detection via Vision-Language Knowledge Distillationcited by 8
  7. 2025Open-Vocabulary HOI Detection with Interaction-aware Prompt and Concept Calibrationcited by 7
  8. 2025Superpowering Open-Vocabulary Object Detectors for X-ray Visioncited by 3
  9. 2025Adapting Vehicle Detectors for Aerial Imagery to Unseen Domains with Weak Supervisioncited by 2
  10. 2025ODOV: Benchmark the Open-Domain Open-Vocabulary Object Detectioncited by 1
  11. 2026A Training-Free Guess What Vision Language Model from Snippets to Open-Vocabulary Object Detectioncited by 1
  12. 2025FedMGP: Personalized Federated Learning with Multi-Group Text-Visual Promptscited by 0
7 more
  1. 2025Semore: VLM-guided Enhanced Semantic Motion Representations for Visual Reinforcement Learningcited by 0
  2. 2025Auto-Vocabulary 3D Object Detectioncited by 0
  3. 2026ObjEmbed: Towards Universal Multimodal Object Embeddingscited by 0
  4. 2026GUIDED: Granular Understanding via Identification, Detection, and Discrimination for Fine-Grained Open-Vocabulary Object Detectioncited by 0
  5. 2026Urban Risk-Aware Navigation via VQA-Based Event Maps for People with Low Visioncited by 0
  6. 2026DSAA: Dual-Stage Attribute Activation for Fine-grained Open Vocabulary Detectioncited by 0
  7. 2026Efficient Quantization-Aware Distillation with Cross-Modal Alignment for Edge Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Enhancing Open-Vocabulary Object Detection through Multi-Level Fine-Grained Visual-Language Alignment
  2. 2024OVLW-DETR: Open-Vocabulary Light-Weighted Detection Transformer
  3. 2025Cyclic Contrastive Knowledge Transfer for Open-Vocabulary Object Detection
  4. 2023DST-Det: Simple Dynamic Self-Training for Open-Vocabulary Object Detection
  5. 2024LLMs Meet VLMs: Boost Open Vocabulary Object Detection with Fine-grained Descriptors
  6. 2026VL-DINO: Leveraging CLIP Vision-Language Knowledge for Open-Vocabulary Object Detectio
  7. 2023Towards Open Vocabulary Learning: A Survey
  8. 2022F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Models