mlsanityData to 14 Oct 2026

LLMDet: Learning Strong Open-Vocabulary Object Detectors under the Supervision of Large Language Models

arXiv 2501.18954 ↗First posted 31 Jan 2025cs.CVcited by 57

Not on the map. Why?

Abstract (arXiv)

Recent open-vocabulary detectors achieve promising performance with abundant region-level annotated data. In this work, we show that an open-vocabulary detector co-training with a large language model by generating image-level detailed captions for each image can further improve performance. To achieve the goal, we first collect a dataset, GroundingCap-1M, wherein each image is accompanied by associated grounding labels and an image-level detailed caption. With this dataset, we finetune an open-vocabulary detector with training objectives including a standard grounding loss and a caption generation loss. We take advantage of a large language model to generate both region-level short captions for each region of interest and image-level long captions for the whole image. Under the supervision of the large language model, the resulting detector, LLMDet, outperforms the baseline by a clear margin, enjoying superior open-vocabulary ability. Further, we show that the improved LLMDet can in turn build a stronger large multi-modal model, achieving mutual benefits. The code, model, and dataset is available at https://github.com/iSEE-Laboratory/LLMDet.

Key facts. Cited by 57 papers in the map (data through 14 Oct 2026). Builds on 58 of its 68 references that are in the map. First posted on arXiv 31 Jan 2025.

Builds on

58 of 68 references.

References · 58

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2020End-to-End Object Detection with Transformerscited by 7,414
  7. 2017Mask R-CNNcited by 7,277
  8. 2024GPT-4o System Cardcited by 4,588
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  10. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  11. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  12. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
38 more
  1. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  2. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  3. 2024Qwen2 Technical Reportcited by 2,120
  4. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  5. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  6. 2019MMDetection: Open MMLab Detection Toolbox and Benchmarkcited by 1,385
  7. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  8. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,319
  9. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,217
  10. 2021Grounded Language-Image Pre-trainingcited by 1,156
  11. 2016Modeling Context in Referring Expressionscited by 1,063
  12. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  13. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 906
  14. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  15. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  16. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 819
  17. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 614
  18. 2024YOLO-World: Real-Time Open-Vocabulary Object Detectioncited by 503
  19. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 357
  20. 2023Scaling Open-Vocabulary Object Detectioncited by 343
  21. 2022GLIPv2: Unifying Localization and Vision-Language Understandingcited by 275
  22. 2023GRES: Generalized Referring Expression Segmentationcited by 254
  23. 2022DetCLIP: Dictionary-Enriched Visual-Concept Paralleled Pre-training for Open-world Detectioncited by 176
  24. 2022F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Modelscited by 150
  25. 2022ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Modelscited by 141
  26. 2024Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoderscited by 139
  27. 2023HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Datacited by 126
  28. 2022GRiT: A Generative Region-to-text Transformer for Object Understandingcited by 121
  29. 2023Aligning Bag of Regions for Open-Vocabulary Object Detectioncited by 119
  30. 2022Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbonecited by 109
  31. 2024T-Rex2: Towards Generic Object Detection via Text-Visual Prompt Synergycited by 94
  32. 2023DetCLIPv2: Scalable Open-Vocabulary Object Detection Pre-training via Word-Region Alignmentcited by 90
  33. 2024The All-Seeing Project V2: Towards General Relation Comprehension of the Open Worldcited by 90
  34. 2024MoVA: Adapting Mixture of Vision Experts to Multimodal Contextcited by 83
  35. 2023V3Det: Vast Vocabulary Visual Detection Datasetcited by 81
  36. 2024An Open and Comprehensive Pipeline for Unified Object Grounding and Detectioncited by 65
  37. 2021Evaluating Large-Vocabulary Object Detectors: The Devil is in the Detailscited by 57
  38. 2024DetCLIPv3: Towards Versatile Generative Open-vocabulary Object Detectioncited by 48

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 25; 2026: 32.

57 papers.

Sort:

By area:

  1. 2025RF-DETR: Neural Architecture Search for Real-Time Detection Transformerscited by 75
  2. 2026Xiaomi-Robotics-0: An Open-Sourced Vision-Language-Action Model with Real-Time Executioncited by 45
  3. 2026Image Generators are Generalist Vision Learnerscited by 43
  4. 2025ViSpeak: Visual Instruction Feedback in Streaming Videoscited by 42
  5. 2025Reinforced Visual Perception with Toolscited by 38
  6. 2026LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decodingcited by 28
  7. 2025Object Detection with Multimodal Large Vision-Language Models: An In-depth Reviewcited by 14
  8. 2025WeDetect: Fast Open-Vocabulary Object Detection as Retrievalcited by 14
  9. 2025FG-CLIP 2: A Bilingual Fine-grained Vision-Language Alignment Modelcited by 13
  10. 2025KeySG: Hierarchical Keyframe-Based 3D Scene Graphscited by 10
  11. 2025A Hierarchical Semantic Distillation Framework for Open-Vocabulary Object Detectioncited by 9
  12. 2025VG-Refiner: Towards Tool-Refined Referring Grounded Reasoning via Agentic Reinforcement Learningcited by 8
38 more
  1. 2026RC-NF: Robot-Conditioned Normalizing Flow for Real-Time Anomaly Detection in Robotic Manipulationcited by 8
  2. 2025Can Current AI Models Count What We Mean, Not What They See? A Benchmark and Systematic Evaluationcited by 7
  3. 2025The SA-FARI Dataset: Segment Anything in Footage of Animals for Recognition and Identificationcited by 7
  4. 2026Revisiting Embodied Chain-of-Thought for Generalizable Robot Manipulationcited by 7
  5. 2025VL-SAM-V2: Open-World Object Detection with General and Specific Query Fusioncited by 6
  6. 2025Adaptive Guidance Semantically Enhanced via Multimodal LLM for Edge-Cloud Object Detectioncited by 6
  7. 2025ConsistCompose: Unified Multimodal Layout Control for Image Compositioncited by 5
  8. 2025Leveraging Vision-Language Models for Open-Vocabulary Instance Segmentation and Trackingcited by 4
  9. 2025MonitorVLM:A Vision Language Framework for Safety Violation Detection in Mining Operationscited by 4
  10. 2025Top-Down Compression: Revisit Efficient Vision Token Projection for Visual Instruction Tuningcited by 3
  11. 2025RAGSR: Regional Attention Guided Diffusion for Image Super-Resolutioncited by 3
  12. 2026DeCo-DETR: Decoupled Cognition DETR for efficient Open-Vocabulary Object Detectioncited by 3
  13. 2026DetPO: In-Context Learning with Multi-Modal LLMs for Few-Shot Object Detectioncited by 2
  14. 2025CQ-DINO: Mitigating Gradient Dilution via Category Queries for Vast Vocabulary Object Detectioncited by 1
  15. 2025Light-Weight Cross-Modal Enhancement Method with Benchmark Construction for UAV-based Open-Vocabulary Object Detectioncited by 1
  16. 2025MRD: Multi-resolution Retrieval-Detection Fusion for High-Resolution Image Understandingcited by 1
  17. 2025Enhancing Visual Programming for Visual Reasoning via Probabilistic Graphscited by 1
  18. 2026CountEx: Fine-Grained Counting via Exemplars and Exclusioncited by 1
  19. 2026ForestPersons: A Large-Scale Dataset for Under-Canopy Missing Person Detectioncited by 1
  20. 2026Prompt-Free Universal Region Proposal Networkcited by 1
  21. 2026Hierarchical and Holistic Open-Vocabulary Functional 3D Scene Graphs for Indoor Spacescited by 1
  22. 2026SPARC: Reliable Spatial Annotations from Robot Demonstrations at Scalecited by 1
  23. 2026The Label Imitation Game: Turing Test Network for Zero-Shot Pseudo-Label Pruningcited by 1
  24. 2026TMI: Text-to-Image Meets Image-to-Image for Complementary Data Synthesis to Boost Long-Tailed Instance Segmentationcited by 1
  25. 2026DeforM: Reasoning-Guided Physics-Aware Video Generation via Spatial-Temporal Maskingcited by 1
  26. 2025ZERO: Industry-ready Vision Foundation Model with Multi-modal Promptscited by 0
  27. 2025Region-Aware Deformable Convolutionscited by 0
  28. 2025State and Scene Enhanced Prototypes for Weakly Supervised Open-Vocabulary Object Detectioncited by 0
  29. 2026Vision Foundation Models for Domain Generalisable Cross-View Localisation in Planetary Ground-Aerial Robotic Teamscited by 0
  30. 2026Do Open-Vocabulary Detectors Transfer to Aerial Imagery? A Comparative Evaluationcited by 0
  31. 2026LAMP: Lift Image-Editing as General 3D Priors for Open-world Manipulationcited by 0
  32. 2026VL-SAM-v3: Memory-Guided Visual Priors for Open-World Object Detectioncited by 0
  33. 2026Thermal-Det: Language-Guided Cross-Modal Distillation for Open-Vocabulary Thermal Object Detectioncited by 0
  34. 2026DetRefiner: Model-Agnostic Detection Refinement with Feature Fusion Transformercited by 0
  35. 2026SceneParser: Hierarchical Scene Parsing for Visual Semantics Understandingcited by 0
  36. 2026COVD: Continual Open-Vocabulary Object Detection with Novel Concept Injectioncited by 0
  37. 2026Vision-Language Grounding as Bidirectional Concept Correspondencecited by 0
  38. 2026Gaze Target Estimation Anywhere with Conceptscited by 0

50 most cited of 57 All 57 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024LLMs Meet VLMs: Boost Open Vocabulary Object Detection with Fine-grained Descriptors
  2. 2023Open-Vocabulary Object Detection using Pseudo Caption Labels
  3. 2025LED: LLM Enhanced Open-Vocabulary Object Detection without Human Curated Data Generation
  4. 2026ExDet: Open-Domain Open-Vocabulary Detection with Cross-modal Extrapolation and Rectification
  5. 2022OmDet: Large-scale vision-language multi-dataset pre-training with multimodal detection network
  6. 2023CoDet: Co-Occurrence Guided Region-Word Alignment for Open-Vocabulary Object Detection
  7. 2026QATMA: Quantization-Aware Training with Multimodal Alignment for Open-Vocabulary Object Detection
  8. 2023Towards Open Vocabulary Learning: A Survey