mlsanityData to 14 Oct 2026

Improving Medical Diagnostics with Vision-Language Models: Convex Hull-Based Uncertainty Analysis

arXiv 2412.00056 ↗First posted 24 Nov 2024cs.CV cs.AIcited by 3

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

In recent years, vision-language models (VLMs) have been applied to various fields, including healthcare, education, finance, and manufacturing, with remarkable performance. However, concerns remain regarding VLMs' consistency and uncertainty, particularly in critical applications such as healthcare, which demand a high level of trust and reliability. This paper proposes a novel approach to evaluate uncertainty in VLMs' responses using a convex hull approach on a healthcare application for Visual Question Answering (VQA). LLM-CXR model is selected as the medical VLM utilized to generate responses for a given prompt at different temperature settings, i.e., 0.001, 0.25, 0.50, 0.75, and 1.00. According to the results, the LLM-CXR VLM shows a high uncertainty at higher temperature settings. Experimental outcomes emphasize the importance of uncertainty in VLMs' responses, especially in healthcare applications.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 21 of its 40 references that are in the map. First posted on arXiv 24 Nov 2024.

Builds on

21 of 40 references.

References · 21

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  7. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  8. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  9. 2019Taking a HINT: Leveraging Explanations to Make Vision and Language Models More Groundedcited by 128
  10. 2020COVID-19 Image Data Collectioncited by 101
  11. 2020Trading Off Diversity and Quality in Natural Language Generationcited by 96
  12. 2023LLM-CXR: Instruction-Finetuned LLM for CXR Image Understanding and Generationcited by 57
9 more
  1. 2023Hot or Cold? Adaptive Temperature Sampling for Code Generation with Large Language Modelscited by 38
  2. 2024Consistency and Uncertainty: Identifying Unreliable Responses From Black-Box Vision-Language Models for Selective Visual Question Answeringcited by 32
  3. 2023DRG-LLaMA : Tuning LLaMA Model to Predict Diagnosis-related Group for Hospitalized Patientscited by 30
  4. 2024Uncertainty-Aware Evaluation for Vision-Language Modelscited by 29
  5. 2023Uncertainty Quantification in Machine Learning for Engineering Design and Health Prognostics: A Tutorialcited by 27
  6. 2015A Survey of Current Datasets for Vision and Language Researchcited by 13
  7. 2024Uncertainty Quantification in Large Language Models Through Convex Hull Analysiscited by 13
  8. 2023LOVM: Language-Only Vision Model Selectioncited by 12
  9. 2014Convex Hulls under Uncertaintycited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Geometric Uncertainty for Detecting and Correcting Hallucinations in LLMscited by 13
  2. 2025DrVD-Bench: Do Vision-Language Models Reason Like Human Doctors in Medical Image Diagnosis?cited by 10
  3. 2026Med-R2: An Adversarial Benchmark for Evidence-Grounded Reasoning in Medical VLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024SURE-VQA: Systematic Understanding of Robustness Evaluation in Medical VQA Tasks
  2. 2026Uncertainty Is Not a Safety Net for Clinical VQA, but Can It Anticipate Model Failure?
  3. 2024Vision-Language Models for Medical Report Generation and Visual Question Answering: A Review
  4. 2025Uncertainty-Driven Expert Control: Enhancing the Reliability of Medical Vision-Language Models
  5. 2026Why Does Grounding Hurt Medical VQA? Benchmarking, Diagnosis, and Fine-Tuning of Vision-Language Models
  6. 2024On Large Visual Language Models for Medical Imaging Analysis: An Empirical Study
  7. 2026Confidence Calibration for Multimodal LLMs: An Empirical Study through Medical VQA
  8. 2026To Agree or To Be Right? The Grounding-Sycophancy Tradeoff in Medical Vision-Language Models