mlsanityData to 7 Oct 2026

Beyond Human Vision: The Role of Large Vision Language Models in Microscope Image Analysis

arXiv 2405.00876 ↗First posted 1 May 2024cs.CV cs.AI cs.LGcited by 7

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Vision language models (VLMs) have recently emerged and gained the spotlight for their ability to comprehend the dual modality of image and textual data. VLMs such as LLaVA, ChatGPT-4, and Gemini have recently shown impressive performance on tasks such as natural image captioning, visual question answering (VQA), and spatial reasoning. Additionally, a universal segmentation model by Meta AI, Segment Anything Model (SAM) shows unprecedented performance at isolating objects from unforeseen images. Since medical experts, biologists, and materials scientists routinely examine microscopy or medical images in conjunction with textual information in the form of captions, literature, or reports, and draw conclusions of great importance and merit, it is indubitably essential to test the performance of VLMs and foundation models such as SAM, on these images. In this study, we charge ChatGPT, LLaVA, Gemini, and SAM with classification, segmentation, counting, and VQA tasks on a variety of microscopy images. We observe that ChatGPT and Gemini are impressively able to comprehend the visual features in microscopy images, while SAM is quite capable at isolating artefacts in a general sense. However, the performance is not close to that of a domain expert - the models are readily encumbered by the introduction of impurities, defects, artefact overlaps and diversity present in the images.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 14 of its 38 references that are in the map. First posted on arXiv 1 May 2024.

Builds on

14 of 38 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024On Large Visual Language Models for Medical Imaging Analysis: An Empirical Study21 Feb
  2. 2023Multimodal ChatGPT for Medical Applications: an Experimental Study of GPT-4V29 Oct
  3. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun

Other references · 11

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2023Segment Anythingcited by 8,158
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2020Image-driven discriminative and generative machine learning algorithms for establishing microstructure-processing relationshipscited by 6
  10. 2021Optimized and autonomous machine learning framework for characterizing pores, particles, grains and grain boundaries in microstructural imagescited by 5
  11. 2021MudrockNet: Semantic Segmentation of Mudrock SEM Images through Deep Learningcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 1.

7 papers.

  1. 2025Can Multimodal LLMs See Materials Clearly? A Multimodal Benchmark on Materials Characterizationcited by 8
  2. 2025Leveraging Vision Capabilities of Multimodal LLMs for Automated Data Extraction from Plotscited by 7
  3. 2025Building Trustworthy Multimodal AI: A Review of Fairness, Transparency, and Ethics in Vision-Language Taskscited by 7
  4. 2026MATRIX: A Multimodal Benchmark and Post-Training Framework for Materials Sciencecited by 3
  5. 2025CubeRobot: Grounding Language in Rubik's Cube Manipulation via Vision-Language Modelcited by 2
  6. 2025On the Domain Robustness of Contrastive Vision-Language Modelscited by 1
  7. 2025In-Context Adaptation of VLMs for Few-Shot Cell Detection in Optical Microscopycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Unifying Segment Anything in Microscopy with Vision-Language Knowledge
  2. 2024μ-Bench: A Vision-Language Benchmark for Microscopy Understanding
  3. 2025MicroVQA++: High-Quality Microscopy Reasoning Dataset with Weakly Supervised Graphs for Multimodal Large Language Model
  4. 2026Benchmarking Vision-Language Models for Microscopic Plant Image Understanding
  5. 2026A Human-in-the-Loop Framework for Efficient Prompt Selection in Microscopy Vision-Language Models
  6. 2025Scaling Large Vision-Language Models for Enhanced Multimodal Comprehension In Biomedical Image Analysis
  7. 2025MicroVQA: A Multimodal Reasoning Benchmark for Microscopy-Based Scientific Research
  8. 2025X-SAM: From Segment Anything to Any Segmentation