mlsanityData to 14 Oct 2026

Vision-Language and Large Language Model Performance in Gastroenterology: GPT, Claude, Llama, Phi, Mistral, Gemma, and Quantized Models

arXiv 2409.00084 ↗First posted 25 Aug 2024cs.CL cs.AIcited by 7

On the map: LLM reasoning → Clinical and biomedical applications

also in 1 more
Abstract (arXiv)

Background and Aims: This study evaluates the medical reasoning performance of large language models (LLMs) and vision language models (VLMs) in gastroenterology. Methods: We used 300 gastroenterology board exam-style multiple-choice questions, 138 of which contain images to systematically assess the impact of model configurations and parameters and prompt engineering strategies utilizing GPT-3.5. Next, we assessed the performance of proprietary and open-source LLMs (versions), including GPT (3.5, 4, 4o, 4omini), Claude (3, 3.5), Gemini (1.0), Mistral, Llama (2, 3, 3.1), Mixtral, and Phi (3), across different interfaces (web and API), computing environments (cloud and local), and model precisions (with and without quantization). Finally, we assessed accuracy using a semiautomated pipeline. Results: Among the proprietary models, GPT-4o (73.7%) and Claude3.5-Sonnet (74.0%) achieved the highest accuracy, outperforming the top open-source models: Llama3.1-405b (64%), Llama3.1-70b (58.3%), and Mixtral-8x7b (54.3%). Among the quantized open-source models, the 6-bit quantized Phi3-14b (48.7%) performed best. The scores of the quantized models were comparable to those of the full-precision models Llama2-7b, Llama2--13b, and Gemma2-9b. Notably, VLM performance on image-containing questions did not improve when the images were provided and worsened when LLM-generated captions were provided. In contrast, a 10% increase in accuracy was observed when images were accompanied by human-crafted image descriptions. Conclusion: In conclusion, while LLMs exhibit robust zero-shot performance in medical reasoning, the integration of visual data remains a challenge for VLMs. Effective deployment involves carefully determining optimal model configurations, encouraging users to consider either the high performance of proprietary models or the flexible adaptability of open-source models.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning. Builds on 7 of its 44 references that are in the map. First posted on arXiv 25 Aug 2024.

Builds on

7 of 44 references.

References · 7

  1. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
  2. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
  3. 2023A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPTcited by 619
  4. 2024Evaluating Quantized Large Language Modelscited by 77
  5. 2024Large Language Models for Medicine: A Surveycited by 38
  6. 2024What Makes Quantization for Large Language Models Hard? An Empirical Study from the Lens of Perturbationcited by 23
  7. 2023DIVKNOWQA: Assessing the Reasoning Ability of LLMs via Open-Domain Question Answering over Knowledge Base and Textcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 4.

7 papers.

  1. 2024LLM-Forest: Ensemble Learning of LLMs with Graph-Augmented Prompts for Data Imputationcited by 25
  2. 2025Kvasir-VQA-x1: A Multimodal Dataset for Medical Reasoning and Robust MedVQA in Gastrointestinal Endoscopycited by 11
  3. 2025Self-Reported Confidence of Large Language Models in Gastroenterology: Analysis of Commercial, Open-Source, and Quantized Modelscited by 3
  4. 2026TCM-DiffRAG: Personalized Syndrome Differentiation Reasoning Method for Traditional Chinese Medicine based on Knowledge Graph and Chain of Thoughtcited by 0
  5. 2026CARE: Privacy-Compliant Agentic Reasoning with Evidence Discordancecited by 0
  6. 2026Uncertainty Is Not a Safety Net for Clinical VQA, but Can It Anticipate Model Failure?cited by 0
  7. 2026Latent Confidence Alignment for LLM Self-Assessmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026GI-Bench: A Panoramic Benchmark Revealing the Knowledge-Experience Dissociation of Multimodal Large Language Models in Gastrointestinal Endoscopy Against Clinical Standards
  2. 2025DL3M: A Vision-to-Language Framework for Expert-Level Medical Reasoning through Deep Learning and Large Language Models
  3. 2026Gastric-X: A Multimodal Multi-Phase Benchmark Dataset for Advancing Vision-Language Models in Gastric Cancer Analysis
  4. 2025Evaluating Large Language Models for Multimodal Simulated Ophthalmic Decision-Making in Diabetic Retinopathy and Glaucoma Screening
  5. 2025How Far Have Medical Vision-Language Models Come? A Comprehensive Benchmarking Study
  6. 2025Hallucination-Aware Multimodal Benchmark for Gastrointestinal Image Analysis with Large Vision-Language Models
  7. 2025DiagR1: A Vision-Language Model Trained via Reinforcement Learning for Digestive Pathology Diagnosis
  8. 2025Performance of GPT-5 Frontier Models in Ophthalmology Question Answering