mlsanityData to 7 Oct 2026

Contextual Object Detection with Multimodal Large Language Models

arXiv 2305.18279 ↗First posted 29 May 2023cs.CV cs.AIcited by 100

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Recent Multimodal Large Language Models (MLLMs) are remarkable in vision-language tasks, such as image captioning and question answering, but lack the essential perception ability, i.e., object detection. In this work, we address this limitation by introducing a novel research problem of contextual object detection -- understanding visible objects within different human-AI interactive contexts. Three representative scenarios are investigated, including the language cloze test, visual captioning, and question answering. Moreover, we present ContextDET, a unified multimodal model that is capable of end-to-end differentiable modeling of visual-language contexts, so as to locate, identify, and associate visual objects with language inputs for human-AI interaction. Our ContextDET involves three key submodels: (i) a visual encoder for extracting visual representations, (ii) a pre-trained LLM for multimodal context decoding, and (iii) a visual decoder for predicting bounding boxes given contextual object words. The new generate-then-detect framework enables us to detect object words within human vocabulary. Extensive experiments show the advantages of ContextDET on our proposed CODE benchmark, open-vocabulary detection, and referring image segmentation. Github: https://github.com/yuhangzang/ContextDET.

Key facts. Cited by 100 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 74 of its 91 references that are in the map. First posted on arXiv 29 May 2023.

Builds on

74 of 91 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr

Other references · 73

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2023GPT-4 Technical Reportcited by 18,132
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  8. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  9. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  10. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  11. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  12. 2023Segment Anythingcited by 8,158
38 more
  1. 2023Visual Instruction Tuningcited by 7,792
  2. 2020End-to-End Object Detection with Transformerscited by 7,384
  3. 2017Mask R-CNNcited by 7,259
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  9. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  10. 2015SSD: Single Shot MultiBox Detectorcited by 3,125
  11. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  12. 2015VQA: Visual Question Answeringcited by 2,489
  13. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  14. 2019FCOS: Fully Convolutional One-Stage Object Detectioncited by 1,573
  15. 2019Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regressioncited by 1,361
  16. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  17. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  18. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,306
  19. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  20. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  21. 2021Grounded Language-Image Pre-trainingcited by 1,147
  22. 2016Modeling Context in Referring Expressionscited by 1,054
  23. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  24. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  25. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  26. 2018CornerNet: Detecting Objects as Paired Keypointscited by 763
  27. 2023DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvaturecited by 705
  28. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  29. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  30. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 613
  31. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 612
  32. 2020Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentationcited by 595
  33. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  34. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  35. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 542
  36. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  37. 2022InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutionscited by 479
  38. 2020Open-Vocabulary Object Detection Using Captionscited by 427

50 most cited of 73 All 73 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 16; 2024: 41; 2025: 33; 2026: 10.

100 papers.

Sort:

By area:

  1. 2025Visual-RFT: Visual Reinforcement Fine-Tuningcited by 501
  2. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  3. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 459
  4. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  5. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  6. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  7. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  8. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  9. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 154
  10. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  11. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understandingcited by 126
  12. 2023Reason2Drive: Towards Interpretable and Chain-based Reasoning for Autonomous Drivingcited by 122
38 more
  1. 2024SIDA: Social Media Image Deepfake Detection, Localization and Explanation with Large Multimodal Modelcited by 107
  2. 2023Embodied Task Planning with Large Language Modelscited by 84
  3. 2024A Survey on Benchmarks of Multimodal Large Language Modelscited by 81
  4. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuningcited by 75
  5. 2025Harmonizing Visual Representations for Unified Multimodal Understanding and Generationcited by 74
  6. 2024Multimodal Alignment and Fusion: A Surveycited by 73
  7. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMscited by 72
  8. 2024GROUNDHOG: Grounding Large Language Models to Holistic Segmentationcited by 70
  9. 2024Many-Shot In-Context Learning in Multimodal Foundation Modelscited by 64
  10. 2023A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Futurecited by 59
  11. 2024LLaRA: Supercharging Robot Learning Data for Vision-Language Policycited by 51
  12. 2023Mitigating Hallucination in Visual Language Models with Visual Supervisioncited by 49
  13. 2023HiLM-D: Enhancing MLLMs with Multi-Scale High-Resolution Details for Autonomous Drivingcited by 48
  14. 2024Learning to Localize Objects Improves Spatial Reasoning in Visual-LLMscited by 48
  15. 2024InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactionscited by 48
  16. 2025UAVs Meet LLMs: Overviews and Perspectives Toward Agentic Low-Altitude Mobilitycited by 48
  17. 2024A Survey on Evaluation of Multimodal Large Language Modelscited by 46
  18. 2024DetToolChain: A New Prompting Paradigm to Unleash Detection Ability of MLLMcited by 45
  19. 2025MM-IFEngine: Towards Multimodal Instruction Followingcited by 41
  20. 2023See, Say, and Segment: Teaching LMMs to Overcome False Premisescited by 40
  21. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  22. 2024Towards Language-Driven Video Inpainting via Multimodal Large Language Modelscited by 33
  23. 2024LTGC: Long-tail Recognition via Leveraging LLMs-driven Generated Contentcited by 33
  24. 2024F-LMM: Grounding Frozen Large Multimodal Modelscited by 33
  25. 2024GenderBias-VL: Benchmarking Gender Bias in Vision Language Models via Counterfactual Probingcited by 31
  26. 2024EMO-LLaMA: Enhancing Facial Emotion Understanding with Instruction Tuningcited by 27
  27. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  28. 2024Exploring the Potential of Large Foundation Models for Open-Vocabulary HOI Detectioncited by 25
  29. 2024A Survey on Large Language Models for Communication, Network, and Service Management: Application Insights, Challenges, and Future Directionscited by 25
  30. 2023Model-as-a-Service (MaaS): A Surveycited by 24
  31. 2023MarineGPT: Unlocking Secrets of Ocean to the Publiccited by 23
  32. 2024VLM-PL: Advanced Pseudo Labeling Approach for Class Incremental Object Detection via Vision-Language Modelcited by 23
  33. 2024Delving into Multi-modal Multi-task Foundation Models for Road Scene Understanding: From Learning Paradigm Perspectivescited by 22
  34. 2024RH20T-P: A Primitive-Level Robotic Dataset Towards Composable Generalization Agentscited by 20
  35. 2024Interpreting Object-level Foundation Models via Visual Precision Searchcited by 20
  36. 2023Large Multimodal Models: Notes on CVPR 2023 Tutorialcited by 18
  37. 2024Leopard: A Vision Language Model For Text-Rich Multi-Image Taskscited by 18
  38. 2024CREMA: Generalizable and Efficient Video-Language Reasoning via Multimodal Modular Fusioncited by 17

50 most cited of 100 All 100 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026FindIt: A Format-Informed Visual Detection Benchmark for Generalist Multimodal LLMs
  2. 2022OmDet: Large-scale vision-language multi-dataset pre-training with multimodal detection network
  3. 2025MQADet: A Plug-and-Play Paradigm for Enhancing Open-Vocabulary Object Detection via Multimodal Question Answering
  4. 2024Detecting Multimodal Situations with Insufficient Context and Abstaining from Baseless Predictions
  5. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
  6. 2023Enhancing HOI Detection with Contextual Cues from Large Vision-Language Models
  7. 2025ORIC: Benchmarking Object Recognition under Contextual Incongruity in Large Vision-Language Models
  8. 2024CL-HOI: Cross-Level Human-Object Interaction Distillation from Vision Large Language Models