A Systematic Analysis of Declining Medical Safety Messaging in Generative AI Models
On the map: Multimodal LLMs / vision-language models → Medical vision-language models
Abstract (arXiv)
Generative AI models, including large language models (LLMs) and vision-language models (VLMs), are increasingly used to interpret medical images and answer clinical questions. Their responses often include inaccuracies; therefore, safety measures like medical disclaimers are critical to remind users that AI outputs are not professionally vetted or a substitute for medical advice. This study evaluated the presence of disclaimers in LLM and VLM outputs across model generations from 2022 to 2025. Using 500 mammograms, 500 chest X-rays, 500 dermatology images, and 500 medical questions, outputs were screened for disclaimer phrases. Medical disclaimer presence in LLM and VLM outputs dropped from 26.3% in 2022 to 0.97% in 2025, and from 19.6% in 2023 to 1.05% in 2025, respectively. By 2025, the majority of models displayed no disclaimers. As public models become more capable and authoritative, disclaimers must be implemented as a safeguard adapting to the clinical context of each output.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 4 of its 19 references that are in the map. First posted on arXiv 8 Jul 2025.
Builds on
4 of 19 references.
References · 4
- 2022Large Language Models Encode Clinical Knowledgecited by 1,807
- 2024The Need for Guardrails with Large Language Models in Medical Safety-Critical Settings: An Artificial Intelligence Application in the Pharmacovigilance Ecosystemcited by 11
- 2023Medical Misinformation in AI-Assisted Self-Diagnosis: Development of a Method (EvalPrompt) for Analyzing Large Language Modelscited by 8
- 2024People over trust AI-generated medical responses and view them to be as valid as doctors, despite low accuracycited by 7
Built on by
Citing papers per year: 2025: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2024Ensuring Safety and Trust: Analyzing the Risks of Large Language Models in Medicine
- 2026Measuring Epistemic Resilience of LLMs Under Misleading Medical Context
- 2025Susceptibility of Large Language Models to User-Driven Factors in Medical Queries
- 2024CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Models
- 2024The Minimum Information about CLinical Artificial Intelligence Checklist for Generative Modeling Research (MI-CLAIM-GEN)
- 2026MedVIGIL: Evaluating Trustworthy Medical VLMs Under Broken Visual Evidence
- 2026Safety and accuracy follow different scaling laws in clinical large language models
- 2026When Medical Safety Alignment Fails: A Benchmark for Evaluating LLMs on High-Risk Medical Queries