mlsanityData to 7 Oct 2026

DetToolChain: A New Prompting Paradigm to Unleash Detection Ability of MLLM

arXiv 2403.12488 ↗First posted 19 Mar 2024cs.CV cs.AIcited by 45

On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection

Abstract (arXiv)

We present DetToolChain, a novel prompting paradigm, to unleash the zero-shot object detection ability of multimodal large language models (MLLMs), such as GPT-4V and Gemini. Our approach consists of a detection prompting toolkit inspired by high-precision detection priors and a new Chain-of-Thought to implement these prompts. Specifically, the prompts in the toolkit are designed to guide the MLLM to focus on regional information (e.g., zooming in), read coordinates according to measure standards (e.g., overlaying rulers and compasses), and infer from the contextual information (e.g., overlaying scene graphs). Building upon these tools, the new detection chain-of-thought can automatically decompose the task into simple subtasks, diagnose the predictions, and plan for progressive box refinements. The effectiveness of our framework is demonstrated across a spectrum of detection tasks, especially hard cases. Compared to existing state-of-the-art methods, GPT-4V with our DetToolChain improves state-of-the-art object detectors by +21.5% AP50 on MS COCO Novel class set for open-vocabulary detection, +24.23% Acc on RefCOCO val set for zero-shot referring expression comprehension, +14.5% AP on D-cube describe object detection FULL setting.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 66 of its 79 references that are in the map. First posted on arXiv 19 Mar 2024.

Builds on

66 of 79 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Aligning Bag of Regions for Open-Vocabulary Object Detection27 Feb

Other references · 65

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  7. 2023Segment Anythingcited by 8,158
  8. 2020End-to-End Object Detection with Transformerscited by 7,384
  9. 2017Mask R-CNNcited by 7,259
  10. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  11. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  12. 2023Qwen Technical Reportcited by 3,273
38 more
  1. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  2. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  3. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  4. 2016Modeling Context in Referring Expressionscited by 1,054
  5. 2023Teaching Large Language Models to Self-Debugcited by 963
  6. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  7. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  8. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  9. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 873
  10. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  11. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
  12. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  13. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 684
  14. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 679
  15. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  16. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  17. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 612
  18. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  19. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  20. 2020Sparse R-CNN: End-to-End Object Detection with Learnable Proposalscited by 494
  21. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 427
  22. 2020Open-Vocabulary Object Detection Using Captionscited by 427
  23. 2022DiffusionDet: Diffusion Model for Object Detectioncited by 354
  24. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 352
  25. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 291
  26. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
  27. 2022DETRs with Collaborative Hybrid Assignments Trainingcited by 273
  28. 2022Open-Vocabulary DETR with Conditional Matchingcited by 220
  29. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 215
  30. 2023Universal Instance Perception as Object Discovery and Retrievalcited by 190
  31. 2018Zero-Shot Object Detectioncited by 189
  32. 2021CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Modelscited by 182
  33. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  34. 2023Compositional Chain-of-Thought Prompting for Large Multimodal Modelscited by 160
  35. 2023CORA: Adapting CLIP for Open-Vocabulary Detection with Region Prompting and Anchor Pre-Matchingcited by 133
  36. 2023DDCoT: Duty-Distinct Chain-of-Thought Prompting for Multimodal Reasoning in Language Modelscited by 124
  37. 2022ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehensioncited by 119
  38. 2023BuboGPT: Enabling Visual Grounding in Multi-Modal LLMscited by 114

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 28; 2026: 9.

45 papers.

Sort:

By area:

  1. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 193
  2. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 176
  3. 2024Number it: Temporal Grounding Videos like Flipping Mangacited by 65
  4. 2024Many-Shot In-Context Learning in Multimodal Foundation Modelscited by 64
  5. 2024Image-of-Thought Prompting for Visual Reasoning Refinement in Multimodal Large Language Modelscited by 64
  6. 2024Visual Agents as Fast and Slow Thinkerscited by 53
  7. 2024Visual Prompting in Multimodal Large Language Models: A Surveycited by 52
  8. 2025Improved Visual-Spatial Reasoning via R1-Zero-Like Trainingcited by 51
  9. 2024TextCoT: Zoom In for Enhanced Multimodal Text-Rich Image Understandingcited by 37
  10. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  11. 2025From Seeing to Doing: Bridging Reasoning and Decision for Robotic Manipulationcited by 34
  12. 2025MMRC: A Large-Scale Benchmark for Understanding Multimodal Large Language Model in Real-World Conversationcited by 28
33 more
  1. 2025Beyond Seeing: Evaluating Multimodal LLMs on Tool-Enabled Image Perception, Transformation, and Reasoningcited by 28
  2. 2026Think3D: Thinking with Space for Spatial Reasoningcited by 22
  3. 2025Prompt-based Adaptation in Large-scale Vision Models: A Surveycited by 21
  4. 2025RSVP: Reasoning Segmentation via Visual Prompting and Multi-modal Chain-of-Thoughtcited by 19
  5. 2024VipAct: Visual-Perception Enhancement via Specialized VLM Agent Collaboration and Tool-usecited by 17
  6. 2025Visual Position Prompt for MLLM based Visual Groundingcited by 16
  7. 2025ONLY: One-Layer Intervention Sufficiently Mitigates Hallucinations in Large Vision-Language Modelscited by 15
  8. 2025How Well Does GPT-4o Understand Vision? Evaluating Multimodal Foundation Models on Standard Computer Vision Taskscited by 14
  9. 2026How Well Do Models Follow Visual Instructions? VIBE: A Systematic Benchmark for Visual Instruction-Driven Image Editingcited by 12
  10. 2025PixelCraft: A Multi-Agent System for High-Fidelity Visual Reasoning on Structured Imagescited by 11
  11. 2025Lost in Time: Clock and Calendar Understanding Challenges in Multimodal LLMscited by 10
  12. 2025From Illusion to Intention: Visual Rationale Learning for Vision-Language Reasoningcited by 9
  13. 2025Octopus: Agentic Multimodal Reasoning with Six-Capability Orchestrationcited by 8
  14. 2024MageBench: Bridging Large Multimodal Models to Agentscited by 6
  15. 20253DAxisPrompt: Promoting the 3D Grounding and Reasoning in GPT-4ocited by 6
  16. 2025MangaVQA and MangaLMM: A Benchmark and Specialized Model for Multimodal Manga Understandingcited by 6
  17. 2025PostAlign: Multimodal Grounding as a Corrective Lens for MLLMscited by 5
  18. 2025Segmentation as A Plug-and-Play Capability for Frozen Multimodal LLMscited by 5
  19. 2025VICoT-Agent: A Vision-Interleaved Chain-of-Thought Framework for Interpretable Multimodal Reasoning and Scalable Remote Sensing Analysiscited by 4
  20. 2025Thinking with Images via Self-Calling Agentcited by 4
  21. 2025VoCap: Video Object Captioning and Segmentation from Any Promptcited by 3
  22. 2025MapTrace: Scalable Data Generation for Route Tracing on Mapscited by 3
  23. 2026Adaptive Time Series Reasoning via Segment Selectioncited by 3
  24. 2026DIVA-GRPO: Enhancing Multimodal Reasoning through Difficulty-Adaptive Variant Advantagecited by 3
  25. 2025OrderChain: Towards General Instruct-Tuning for Stimulating the Ordinal Understanding Ability of MLLMcited by 2
  26. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Surveycited by 2
  27. 2025ToolScope: An Agentic Framework for Vision-Guided and Long-Horizon Tool Usecited by 2
  28. 2026Chain-of-Caption: Training-free improvement of multimodal large language model on referring expression comprehensioncited by 1
  29. 2025MIND: Multi-rationale INtegrated Discriminative Reasoning Framework for Multi-modal Large Modelscited by 0
  30. 2026Micro-Macro Retrieval: Reducing Long-Form Hallucination in Large Language Modelscited by 0
  31. 2026From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Modelscited by 0
  32. 2026Multimodal Plant Root Phenotyping with Integration of 3D Skeleton Extraction and Language Analysiscited by 0
  33. 2026presto: Efficient, Training-free, and Open-world Object Placement via Imaginary Searchcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026FindIt: A Format-Informed Visual Detection Benchmark for Generalist Multimodal LLMs
  2. 2026Is It Time for the Renaissance of Salient Object Detection in the Era of MLLMs?
  3. 2026DetPO: In-Context Learning with Multi-Modal LLMs for Few-Shot Object Detection
  4. 2025Object Detection with Multimodal Large Vision-Language Models: An In-depth Review
  5. 2025An Empirical Analysis of VLM-based OOD Detection: Mechanisms, Advantages, and Sensitivity
  6. 2026A Training-Free Guess What Vision Language Model from Snippets to Open-Vocabulary Object Detection
  7. 2025Vision-Language Model for Object Detection and Segmentation: A Review and Evaluation
  8. 2025LMM-Det: Make Large Multimodal Models Excel in Object Detection