mlsanityData to 14 Oct 2026

Object Detection with Multimodal Large Vision-Language Models: An In-depth Review

arXiv 2508.19294 ↗First posted 25 Aug 2025cs.CV cs.AI cs.CLcited by 14

On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection

Abstract (arXiv)

The fusion of language and vision in large vision-language models (LVLMs) has revolutionized deep learning-based object detection by enhancing adaptability, contextual reasoning, and generalization beyond traditional architectures. This in-depth review presents a structured exploration of the state-of-the-art in LVLMs, systematically organized through a three-step research review process. First, we discuss the functioning of vision language models (VLMs) for object detection, describing how these models harness natural language processing (NLP) and computer vision (CV) techniques to revolutionize object detection and localization. We then explain the architectural innovations, training paradigms, and output flexibility of recent LVLMs for object detection, highlighting how they achieve advanced contextual understanding for object detection. The review thoroughly examines the approaches used in integration of visual and textual information, demonstrating the progress made in object detection using VLMs that facilitate more sophisticated object detection and localization strategies. This review presents comprehensive visualizations demonstrating LVLMs' effectiveness in diverse scenarios including localization and segmentation, and then compares their real-time performance, adaptability, and complexity to traditional deep learning systems. Based on the review, its is expected that LVLMs will soon meet or surpass the performance of conventional methods in object detection. The review also identifies a few major limitations of the current LVLM modes, proposes solutions to address those challenges, and presents a clear roadmap for the future advancement in this field. We conclude, based on this study, that the recent advancement in LVLMs have made and will continue to make a transformative impact on object detection and robotic applications in the future.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 229 of its 348 references that are in the map. First posted on arXiv 25 Aug 2025.

Builds on

229 of 348 references.

Closest ancestors · 18

Same direction, newest first

  1. 2025Roboflow100-VL: A Multi-Domain Object Detection Benchmark for Vision-Language Models27 May
  2. 2025TS-VLM: Text-Guided SoftSort Pooling for Vision-Language Models in Multi-View Driving Reasoning19 May
  3. 2025Cyclic Contrastive Knowledge Transfer for Open-Vocabulary Object Detection14 Mar
  4. 2025The Power of One: A Single Example is All it Takes for Segmentation in VLMs13 Mar
  5. 2025CurricuVLM: Towards Safe Autonomous Driving via Personalized Safety-Critical Curriculum Learning with Vision-Language Models21 Feb
  6. 2025Zero-Shot Scene Understanding for Automatic Target Recognition Using Large Vision-Language Models13 Jan
  7. 2024Integrating Object Detection Modality into Visual Language Model for Enhanced Autonomous Driving Agent8 Nov
  8. 2024Advancing Object Detection in Transportation with Multimodal Large Language Models (MLLMs): A Comprehensive Review and Empirical Testing26 Sep
  9. 2024VLMine: Long-Tail Data Mining with Vision Language Models23 Sep
  10. 2024Retrieval-Augmented Open-Vocabulary Object Detection8 Apr
  11. 2024TaskCLIP: Extend Large Vision-Language Model for Task Oriented Object Detection12 Mar
  12. 2024VLM-PL: Advanced Pseudo Labeling Approach for Class Incremental Object Detection via Vision-Language Model8 Mar
6 more
  1. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models19 Feb
  2. 2024LLMs Meet VLMs: Boost Open Vocabulary Object Detection with Fine-grained Descriptors7 Feb
  3. 2023Revisiting Few-Shot Object Detection with Vision-Language Models22 Dec
  4. 2023Chain of Visual Perception: Harnessing Multimodal Large Language Models for Zero-shot Camouflaged Object Detection19 Nov
  5. 2023Aligning Bag of Regions for Open-Vocabulary Object Detection27 Feb
  6. 2022Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Model28 Mar

Other references · 211

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  3. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2017Mask R-CNNcited by 7,277
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  8. 2025Qwen2.5-VL Technical Reportcited by 5,587
  9. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,852
  10. 2024GPT-4o System Cardcited by 4,588
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  12. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
38 more
  1. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  2. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  4. 2015SSD: Single Shot MultiBox Detectorcited by 3,130
  5. 2015Fast R-CNNcited by 2,933
  6. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  7. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  8. 2017Cascade R-CNN: Delving into High Quality Object Detectioncited by 1,276
  9. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,228
  10. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,225
  11. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,217
  12. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
  13. 2023PaLM 2 Technical Reportcited by 1,162
  14. 2021Grounded Language-Image Pre-trainingcited by 1,156
  15. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  16. 2019EfficientDet: Scalable and Efficient Object Detectioncited by 1,020
  17. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 906
  18. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  19. 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 812
  20. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 724
  21. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  22. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 706
  23. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 681
  24. 2019CenterNet: Keypoint Triplets for Object Detectioncited by 654
  25. 2023A Comprehensive Overview of Large Language Modelscited by 648
  26. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 616
  27. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 584
  28. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 572
  29. 2023DETRs Beat YOLOs on Real-time Object Detectioncited by 533
  30. 2018Object Detection with Deep Learning: A Reviewcited by 528
  31. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Modelcited by 526
  32. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 517
  33. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 509
  34. 2024YOLO-World: Real-Time Open-Vocabulary Object Detectioncited by 503
  35. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  36. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 474
  37. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 455
  38. 2019Cascade R-CNN: High Quality Object Detection and Instance Segmentationcited by 429

50 most cited of 211 All 211 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 10.

14 papers.

Sort:

By area:

  1. 2025YOLO26: Key Architectural Enhancements and Performance Benchmarking for Real-Time Object Detectioncited by 79
  2. 2025MonitorVLM:A Vision Language Framework for Safety Violation Detection in Mining Operationscited by 4
  3. 2026AIVD: Adaptive Edge-Cloud Collaboration for Accurate and Efficient Industrial Visual Detectioncited by 4
  4. 2025Aligning Perception, Reasoning, Modeling and Interaction: A Survey on Physical AIcited by 3
  5. 2025MMDrive: Interactive Scene Understanding Beyond Vision with Multi-representational Fusioncited by 3
  6. 2026YOLOE-26: Integrating YOLO26 with YOLOE for Real-Time Open-Vocabulary Instance Segmentationcited by 3
  7. 2026Tarot-SAM3: Training-free SAM3 for Any Referring Expression Segmentationcited by 1
  8. 2026Adversarial Orthogonal Disentanglement for LVLM Hallucination Mitigationcited by 1
  9. 2026HKVLM: Faithful Query--Region Binding for Frozen-Detector Visual Groundingcited by 1
  10. 2026CP-WSP: A Declarative CP-SAT Framework for Configurable Multi-Constraint Workforce Schedulingcited by 1
  11. 2026Ground-Truth Depth in Vision Language Models: Spatial Context Understanding in Conversational AI for XR-Robotic Support in Emergency First Responsecited by 0
  12. 2026From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Modelscited by 0
2 more
  1. 2026Assessing the Benefits of Combining Advanced Deep Learning Techniques for Post-Disaster Building Damage Assessment from UAV Imagerycited by 0
  2. 2026GeBDA: Building Damage Assessment as Text-Based Sequence Predictioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Vision-Language Model for Object Detection and Segmentation: A Review and Evaluation
  2. 2025A Review of 3D Object Detection with Vision-Language Models
  3. 2021VinVL: Revisiting Visual Representations in Vision-Language Models
  4. 2026FindIt: A Format-Informed Visual Detection Benchmark for Generalist Multimodal LLMs
  5. 2022F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Models
  6. 2024VLM-HOI: Vision Language Models for Interpretable Human-Object Interaction Analysis
  7. 2025Leveraging Vision-Language Models for Open-Vocabulary Instance Segmentation and Tracking
  8. 2022OmDet: Large-scale vision-language multi-dataset pre-training with multimodal detection network