mlsanityData to 7 Oct 2026

Evaluating LLM -- Generated Multimodal Diagnosis from Medical Images and Symptom Analysis

arXiv 2402.01730 ↗First posted 28 Jan 2024cs.CL cs.AI cs.CVcited by 23

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) constitute a breakthrough state-of-the-art Artificial Intelligence technology which is rapidly evolving and promises to aid in medical diagnosis. However, the correctness and the accuracy of their returns has not yet been properly evaluated. In this work, we propose an LLM evaluation paradigm that incorporates two independent steps of a novel methodology, namely (1) multimodal LLM evaluation via structured interactions and (2) follow-up, domain-specific analysis based on data extracted via the previous interactions. Using this paradigm, (1) we evaluate the correctness and accuracy of LLM-generated medical diagnosis with publicly available multimodal multiple-choice questions(MCQs) in the domain of Pathology and (2) proceed to a systemic and comprehensive analysis of extracted results. We used GPT-4-Vision-Preview as the LLM to respond to complex, medical questions consisting of both images and text, and we explored a wide range of diseases, conditions, chemical compounds, and related entity types that are included in the vast knowledge domain of Pathology. GPT-4-Vision-Preview performed quite well, scoring approximately 84% of correct diagnoses. Next, we further analyzed the findings of our work, following an analytical approach which included Image Metadata Analysis, Named Entity Recognition and Knowledge Graphs. Weaknesses of GPT-4-Vision-Preview were revealed on specific knowledge paths, leading to a further understanding of its shortcomings in specific areas. Our methodology and findings are not limited to the use of GPT-4-Vision-Preview, but a similar approach can be followed to evaluate the usefulness and accuracy of other LLMs and, thus, improve their use with further optimization.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 9 of its 40 references that are in the map. First posted on arXiv 28 Jan 2024.

Builds on

9 of 40 references.

References · 9

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  4. 2021Transformers in Vision: A Surveycited by 1,009
  5. 2020A Survey on Knowledge Graphs: Representation, Acquisition and Applicationscited by 643
  6. 2021SiT: Self-supervised vIsion Transformercited by 99
  7. 2020Domain-specific Knowledge Graphs: A surveycited by 68
  8. 2023Advances in Medical Image Analysis with Vision Transformers: A Comprehensive Reviewcited by 62
  9. 2022Fine-tuning Image Transformers using Learnable Memorycited by 34

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 11; 2026: 3.

23 papers.

Sort:

By area:

  1. 2025Agentic Large Language Models, a surveycited by 105
  2. 2024Synergizing Foundation Models and Federated Learning: A Surveycited by 14
  3. 2024A Survey of Medical Vision-and-Language Applications and Their Techniquescited by 11
  4. 2025Hallucination at a Glance: Controlled Visual Edits and Fine-Grained Multimodal Learningcited by 11
  5. 2025Leveraging Vision Capabilities of Multimodal LLMs for Automated Data Extraction from Plotscited by 7
  6. 2025MedOmni-45deg: A Safety-Performance Benchmark for Reasoning-Oriented LLMs in Medicinecited by 6
  7. 2024Dermacen Analytica: A Novel Methodology Integrating Multi-Modal Large Language Models with Machine Learning in tele-dermatologycited by 4
  8. 2024COGNET-MD, an evaluation framework and dataset for Large Language Model benchmarks in the medical domaincited by 4
  9. 2025Advanced Deep Learning and Large Language Models: Comprehensive Insights for Cancer Detectioncited by 4
  10. 20243DS: Medical Domain Adaptation of LLMs via Decomposed Difficulty-based Data Selectioncited by 3
  11. 2024MpoxVLM: A Vision-Language Model for Diagnosing Skin Lesions from Mpox Virus Infectioncited by 3
  12. 2025GENUINE: Graph Enhanced Multi-level Uncertainty Estimation for Large Language Modelscited by 3
11 more
  1. 2024Watermarking Large Language Models and the Generated Content: Opportunities and Challengescited by 2
  2. 2024Reliable Poisoned Sample Detection against Backdoor Attacks Enhanced by Sharpness Aware Minimizationcited by 2
  3. 2025Trustworthy Reasoning: Evaluating and Enhancing Factual Accuracy in LLM Intermediate Thought Processescited by 1
  4. 2025Large language models surpass domain-specific architectures for antepartum electronic fetal monitoring analysiscited by 1
  5. 2025ClinStructor: AI-Powered Structuring of Unstructured Clinical Textscited by 1
  6. 2024An LLM-based Simulation Framework for Embodied Conversational Agents in Psychological Counselingcited by 0
  7. 2025FedSODA: Federated Fine-tuning of LLMs via Similarity Group Pruning and Orchestrated Distillation Alignmentcited by 0
  8. 2025CataractSurg-80K: Knowledge-Driven Benchmarking for Structured Reasoning in Ophthalmic Surgery Planningcited by 0
  9. 2026IMCBench: A benchmark for multimodal LLMs in Image-grounded Medical Conversationscited by 0
  10. 2026SABER: A Semantic-Aligned Brain Network Analysis Framework via Multi-scale Hypergraphscited by 0
  11. 2026Sense and Sensitivity: Benchmarking LLM Clinical Triage Recommendations with Physician Expertscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026MedReaMM: Evaluating Large Multimodal Models on Expert-Level Clinical Diagnostic Synthesis
  2. 2024A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicine
  3. 2026LLM is Not All You Need: A Systematic Evaluation of ML vs. Foundation Models for text and image based Medical Classification
  4. 2024Worse than Random? An Embarrassingly Simple Probing Evaluation of Large Multimodal Models in Medical VQA
  5. 2023ChatCAD: Interactive Computer-Aided Diagnosis on Medical Image using Large Language Models
  6. 2024GMAI-VL & GMAI-VL-5.5M: A Large Vision-Language Model and A Comprehensive Multimodal Dataset Towards General Medical AI
  7. 2026MMRareBench: A Rare-Disease Multimodal and Multi-Image Medical Benchmark
  8. 2025A Medical Multimodal Diagnostic Framework Integrating Vision-Language Models and Logic Tree Reasoning