mlsanityData to 7 Oct 2026

Effectiveness Assessment of Recent Large Vision-Language Models

arXiv 2403.04306 ↗First posted 7 Mar 2024cs.CV cs.AI cs.LGcited by 29

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

The advent of large vision-language models (LVLMs) represents a remarkable advance in the quest for artificial general intelligence. However, the model's effectiveness in both specialized and general tasks warrants further investigation. This paper endeavors to evaluate the competency of popular LVLMs in specialized and general tasks, respectively, aiming to offer a comprehensive understanding of these novel models. To gauge their effectiveness in specialized tasks, we employ six challenging tasks in three different application scenarios: natural, healthcare, and industrial. These six tasks include salient/camouflaged/transparent object detection, as well as polyp detection, skin lesion detection, and industrial anomaly detection. We examine the performance of three recent open-source LVLMs, including MiniGPT-v2, LLaVA-1.5, and Shikra, on both visual recognition and localization in these tasks. Moreover, we conduct empirical investigations utilizing the aforementioned LVLMs together with GPT-4V, assessing their multi-modal understanding capabilities in general tasks including object counting, absurd question answering, affordance reasoning, attribute recognition, and spatial relation reasoning. Our investigations reveal that these LVLMs demonstrate limited proficiency not only in specialized tasks but also in general tasks. We delve deep into this inadequacy and uncover several potential factors, including limited cognition in specialized tasks, object hallucination, text-to-image interference, and decreased robustness in complex problems. We hope that this study can provide useful insights for the future development of LVLMs, helping researchers improve LVLMs for both general and specialized applications.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 101 of its 143 references that are in the map. First posted on arXiv 7 Mar 2024.

Builds on

101 of 143 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024ViGoR: Improving Visual Grounding of Large Vision Language Models with Fine-Grained Reward Modeling9 Feb
  2. 2023A Challenger to GPT-4V? Early Explorations of Gemini in Visual Expertise19 Dec
  3. 2022Leveraging per Image-Token Consistency for Vision-Language Pre-training20 Nov
  4. 2022Vision-Language Pre-Training with Triple Contrastive Learning21 Feb
  5. 2022VLP: A Survey on Vision-Language Pre-training18 Feb
  6. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan

Other references · 95

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  6. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  7. 2023GPT-4 Technical Reportcited by 18,132
  8. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  9. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  10. 2023Segment Anythingcited by 8,158
  11. 2023Visual Instruction Tuningcited by 7,792
  12. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
38 more
  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  2. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  3. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  4. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  5. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  6. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  7. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  8. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  9. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  10. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  11. 2017Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), Hosted by the International Skin Imaging Collaboration (ISIC)cited by 549
  12. 2017Adversarial Patchcited by 514
  13. 2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 488
  14. 2017Structure-measure: A New Way to Evaluate Foreground Mapscited by 480
  15. 2022SPot-the-Difference Self-Supervised Pre-training for Anomaly Detection and Segmentationcited by 397
  16. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Modelscited by 314
  17. 2019Fooling automated surveillance cameras: adversarial patches to attack person detectioncited by 268
  18. 2019Visual Entailment: A Novel Task for Fine-Grained Image Understandingcited by 250
  19. 2023MOSE: A New Dataset for Video Object Segmentation in Complex Scenescited by 232
  20. 2023MeViS: A Large-scale Benchmark for Video Segmentation with Motion Expressionscited by 225
  21. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
  22. 2021Concealed Object Detectioncited by 212
  23. 2016Skin Lesion Analysis toward Melanoma Detection: A Challenge at the International Symposium on Biomedical Imaging (ISBI) 2016, hosted by the International Skin Imaging Collaboration (ISIC)cited by 197
  24. 2023AnomalyGPT: Detecting Industrial Anomalies Using Large Vision-Language Modelscited by 169
  25. 2018DPatch: An Adversarial Patch Attack on Object Detectorscited by 146
  26. 2022Towards Adversarial Attack on Vision-Language Pre-training Modelscited by 142
  27. 2023Holistic Analysis of Hallucination in GPT-4V(ision): Bias and Interference Challengescited by 131
  28. 2022Language Models with Image Descriptors are Strong Few-Shot Video-Language Learnerscited by 117
  29. 2023BadCLIP: Dual-Embedding Guided Backdoor Attack on Multimodal Contrastive Learningcited by 112
  30. 2018Transferable Adversarial Attacks for Image and Video Object Detectioncited by 109
  31. 2023Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Modelscited by 106
  32. 2019Learning to Compose and Reason with Language Tree Structures for Visual Groundingcited by 98
  33. 2020Segmenting Transparent Objects in the Wildcited by 98
  34. 2019On Physical Adversarial Patches for Object Detectioncited by 93
  35. 2024VL-Trojan: Multimodal Instruction Backdoor Attacks against Autoregressive Visual Language Modelscited by 83
  36. 2020JL-DCF: Joint Learning and Densely-Cooperative Fusion Framework for RGB-D Salient Object Detectioncited by 79
  37. 2022Parallel Rectangle Flip Attack: A Query-based Black-box Attack against Object Detectioncited by 79
  38. 2023SAM Struggles in Concealed Scenes -- Empirical Study on Segment Anythingcited by 75

50 most cited of 95 All 95 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 13; 2026: 8.

29 papers.

Sort:

By area:

  1. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  2. 2024DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Modelscited by 93
  3. 2024KiVA: Kid-inspired Visual Analogies for Testing Large Multimodal Modelscited by 22
  4. 2025Docopilot: Improving Multimodal Models for Document-Level Understandingcited by 19
  5. 2024Grounding-IQA: Grounding Multimodal Language Model for Image Quality Assessmentcited by 12
  6. 2025LoRASculpt: Sculpting LoRA for Harmonizing General and Specialized Knowledge in Multimodal Large Language Modelscited by 11
  7. 2024VLM4Bio: A Benchmark Dataset to Evaluate Pretrained Vision-Language Models for Trait Discovery from Biological Imagescited by 10
  8. 2025Poison as Cure: Visual Noise for Mitigating Object Hallucinations in LVMscited by 9
  9. 2024La-SoftMoE CLIP for Unified Physical-Digital Face Attack Detectioncited by 8
  10. 2025Autoregressive Denoising Score Matching is a Good Video Anomaly Detectorcited by 4
  11. 2024VideoLights: Feature Refinement and Cross-Task Alignment Transformer for Joint Video Highlight Detection and Moment Retrievalcited by 3
  12. 2024Can Large Language Models Grasp Event Signals? Exploring Pure Zero-Shot Event-based Recognitioncited by 2
17 more
  1. 2025Benchmarking Large and Small MLLMscited by 2
  2. 2025A Review on Geometry and Surface Inspection in 3D Concrete Printingcited by 2
  3. 2025TokenFocus-VQA: Enhancing Text-to-Image Alignment with Position-Aware Focus and Multi-Perspective Aggregations on LVLMscited by 2
  4. 2025An Expert Ensemble for Detecting Anomalous Scenes, Interactions, and Behaviors in Autonomous Drivingcited by 1
  5. 2025Robust Diagram Reasoning: A Framework for Enhancing LVLM Performance on Visually Perturbed Scientific Diagramscited by 1
  6. 2025CAD2DMD-SET: Synthetic Generation Tool of Digital Measurement Device CAD Model Datasets for fine-tuning Large Vision-Language Modelscited by 1
  7. 2026Can Vision Language Models Learn Intuitive Physics from Interaction?cited by 1
  8. 2026Attend to Anything: Foundation Model for Unified Human Attention Modelingcited by 1
  9. 2025Unleashing the Power of Motion and Depth: A Selective Fusion Strategy for RGB-D Video Salient Object Detectioncited by 0
  10. 2025MLP-Offload: Multi-Level, Multi-Path Offloading for LLM Pre-training to Break the GPU Memory Wallcited by 0
  11. 2025Table Comprehension in Building Codes using Vision Language Models and Domain-Specific Fine-Tuningcited by 0
  12. 2026Camouflage-aware Image-Text Retrieval via Expert Collaborationcited by 0
  13. 2026DGSSM: Diffusion guided state-space models for multimodal salient object detectioncited by 0
  14. 2026DO-Bench: An Attributable Benchmark for Diagnosing Object Hallucination in Vision-Language Modelscited by 0
  15. 2026A Hybrid Vision-Language Architecture for Automated Defect Reasoning and Report Generation in Industrial Inspectioncited by 0
  16. 2026Failure-Aware Refinement of Vision-Language Model for Lithography Defect Detectioncited by 0
  17. 2026Pay More Attention To Text In High-Resolution MLLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Object Detection with Multimodal Large Vision-Language Models: An In-depth Review
  2. 2025Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation
  3. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challenges
  4. 2026FindIt: A Format-Informed Visual Detection Benchmark for Generalist Multimodal LLMs
  5. 2026A Comparative Evaluation of Large Vision-Language Models for 2D Object Detection under SOTIF Conditions
  6. 2023ReForm-Eval: Evaluating Large Vision Language Models via Unified Re-Formulation of Task-Oriented Benchmarks
  7. 2024Response Wide Shut: Surprising Observations in Basic Vision Language Model Capabilities
  8. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?