mlsanityData to 7 Oct 2026

DetGPT: Detect What You Need via Reasoning

arXiv 2305.14167 ↗First posted 23 May 2023cs.CV cs.AIcited by 112

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

In recent years, the field of computer vision has seen significant advancements thanks to the development of large language models (LLMs). These models have enabled more effective and sophisticated interactions between humans and machines, paving the way for novel techniques that blur the lines between human and machine intelligence. In this paper, we introduce a new paradigm for object detection that we call reasoning-based object detection. Unlike conventional object detection methods that rely on specific object names, our approach enables users to interact with the system using natural language instructions, allowing for a higher level of interactivity. Our proposed method, called DetGPT, leverages state-of-the-art multi-modal models and open-vocabulary object detectors to perform reasoning within the context of the user's instructions and the visual scene. This enables DetGPT to automatically locate the object of interest based on the user's expressed desires, even if the object is not explicitly mentioned. For instance, if a user expresses a desire for a cold beverage, DetGPT can analyze the image, identify a fridge, and use its knowledge of typical fridge contents to locate the beverage. This flexibility makes our system applicable across a wide range of fields, from robotics and automation to autonomous driving. Overall, our proposed paradigm and DetGPT demonstrate the potential for more sophisticated and intuitive interactions between humans and machines. We hope that our proposed paradigm and approach will provide inspiration to the community and open the door to more interative and versatile object detection systems. Our project page is launched at detgpt.github.io.

Key facts. Cited by 112 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 50 of its 72 references that are in the map. First posted on arXiv 23 May 2023.

Builds on

50 of 72 references.

References · 50

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  8. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  9. 2020End-to-End Object Detection with Transformerscited by 7,384
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  11. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  12. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
38 more
  1. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  4. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  5. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  6. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  7. 2015Fast R-CNNcited by 2,930
  8. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  10. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  11. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  12. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  13. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  14. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  15. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  16. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  17. 2021Grounded Language-Image Pre-trainingcited by 1,147
  18. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  19. 2016Bag of Tricks for Efficient Text Classificationcited by 869
  20. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 684
  21. 2019CenterNet: Keypoint Triplets for Object Detectioncited by 654
  22. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  23. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 435
  24. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  25. 2022ZeroGen: Efficient Zero-shot Learning via Dataset Generationcited by 209
  26. 2022Generating Training Data with Language Models: Towards Zero-Shot Language Understandingcited by 199
  27. 2021Generating Datasets with Pretrained Language Modelscited by 192
  28. 2022DetCLIP: Dictionary-Enriched Visual-Concept Paralleled Pre-training for Open-world Detectioncited by 176
  29. 2023Active Prompting with Chain-of-Thought for Large Language Modelscited by 154
  30. 2023Automatic Prompt Augmentation and Selection with Chain-of-Thought from Labeled Datacited by 119
  31. 2020ConvBERT: Improving BERT with Span-based Dynamic Convolutioncited by 88
  32. 2022ProGen: Progressive Zero-shot Dataset Generation via In-context Feedbackcited by 67
  33. 2022Self-Guided Noise-Free Data Generation for Efficient Zero-Shot Learningcited by 49
  34. 2021Support-Set Based Cross-Supervision for Video Groundingcited by 28
  35. 2021G-DetKD: Towards General Distillation Framework for Object Detectors via Contrastive and Semantic-guided Feature Imitationcited by 27
  36. 2021Joint-DetNAS: Upgrade Your Detector with NAS, Pruning and Dynamic Distillationcited by 23
  37. 2021AutoBERT-Zero: Evolving BERT Backbone from Scratchcited by 21
  38. 2022Write and Paint: Generative Vision-Language Models are Unified Modal Learnerscited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 27; 2024: 51; 2025: 28; 2026: 6.

112 papers.

Sort:

By area:

  1. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  2. 2023A Survey on Multimodal Large Language Modelscited by 898
  3. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  4. 2023A Comprehensive Overview of Large Language Modelscited by 639
  5. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 447
  6. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 330
  7. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  8. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  9. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  10. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  11. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  12. 2025Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligencecited by 206
38 more
  1. 2023LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agentscited by 188
  2. 2024MobileVLM V2: Faster and Stronger Baseline for Vision Language Modelcited by 172
  3. 2025WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMscited by 147
  4. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  5. 2024MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scalecited by 125
  6. 2024VISA: Reasoning Video Object Segmentation via Large Language Modelscited by 123
  7. 2023Reason2Drive: Towards Interpretable and Chain-based Reasoning for Autonomous Drivingcited by 122
  8. 2024Groma: Localized Visual Tokenization for Grounding Multimodal Large Language Modelscited by 118
  9. 2024MLLM-Protector: Ensuring MLLM's Safety without Hurting Performancecited by 111
  10. 2024LamRA: Large Multimodal Model as Your Advanced Retrieval Assistantcited by 106
  11. 2024One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videoscited by 99
  12. 2024Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Modelscited by 97
  13. 2023InstructMol: Multi-Modal Integration for Building a Versatile and Reliable Molecular Assistant in Drug Discoverycited by 86
  14. 2023OpenIns3D: Snap and Lookup for 3D Open-vocabulary Instance Segmentationcited by 82
  15. 2024Holistic Autonomous Driving Understanding by Bird's-Eye-View Injected Multi-Modal Large Modelscited by 75
  16. 2025Efficient Reasoning with Hidden Thinkingcited by 72
  17. 2023ChatPose: Chatting about 3D Human Posecited by 71
  18. 2023LISA++: An Improved Baseline for Reasoning Segmentation with Large Language Modelcited by 70
  19. 2024GROUNDHOG: Grounding Large Language Models to Holistic Segmentationcited by 70
  20. 2024SAM4MLLM: Enhance Multi-Modal Large Language Model for Referring Expression Segmentationcited by 68
  21. 2024Multi-Object Hallucination in Vision-Language Modelscited by 60
  22. 2023A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Futurecited by 59
  23. 2023LMFlow: An Extensible Toolkit for Finetuning and Inference of Large Foundation Modelscited by 58
  24. 2024Visual Agents as Fast and Slow Thinkerscited by 53
  25. 2023Jack of All Tasks, Master of Many: Designing General-purpose Coarse-to-Fine Vision-Language Modelcited by 52
  26. 2023Merlin:Empowering Multimodal LLMs with Foresight Mindscited by 46
  27. 2024MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMscited by 38
  28. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  29. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  30. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  31. 2025GEM: Empowering MLLM for Grounded ECG Understanding with Time Series and Imagescited by 35
  32. 2024SegPoint: Segment Any Point Cloud via Large Language Modelcited by 34
  33. 2024Unveiling Visual Perception in Language Models: An Attention Head Analysis Approachcited by 34
  34. 2024Towards Language-Driven Video Inpainting via Multimodal Large Language Modelscited by 33
  35. 2023Tracking with Human-Intent Reasoningcited by 32
  36. 2025ReferDINO: Referring Video Object Segmentation with Visual Grounding Foundationscited by 30
  37. 2023Lenna: Language Enhanced Reasoning Detection Assistantcited by 29
  38. 2024Generative Region-Language Pretraining for Open-Ended Object Detectioncited by 29

50 most cited of 112 All 112 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Open-Vocabulary DETR with Conditional Matching
  2. 2023CoTDet: Affordance Knowledge Prompting for Task Driven Object Detection
  3. 2024DetToolChain: A New Prompting Paradigm to Unleash Detection Ability of MLLM
  4. 2025FLORA: Formal Language Model Enables Robust Training-free Zero-shot Object Referring Analysis
  5. 2023InstructDET: Diversifying Referring Object Detection with Generalized Instructions
  6. 2025Reasoning in Computer Vision: Taxonomy, Models, Tasks, and Methodologies
  7. 2025Open-Det: An Efficient Learning Framework for Open-Ended Detection
  8. 2023CLIP the Gap: A Single Domain Generalization Approach for Object Detection