# Ensuring Safety and Trust: Analyzing the Risks of Large Language Models in Medicine

arXiv: https://arxiv.org/abs/2411.14487 · HTML: https://mlsanity.com/paper/2411.14487/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security.
- First posted on arXiv 20 Nov 2024.

## Placed in

- LLM safety and security / [Safety alignment, refusal and guardrails](https://mlsanity.com/safety/safety-alignment-refusal-and-guardrails/)

## Built on by (most cited first, 7 of 7)

- [Shallow Robustness, Deep Vulnerabilities: Multi-Turn Evaluation of Medical LLMs](https://mlsanity.com/paper/2510.12255/) (2025)
- [MPIB: A Benchmark for Medical Prompt Injection Attacks and Clinical Safety in LLMs](https://mlsanity.com/paper/2602.06268/) (2026)
- [Constrained Sliced Wasserstein Embedding](https://mlsanity.com/paper/2506.02203/) (2025)
- [Gene-R1: Reasoning with Data-Augmented Lightweight LLMs for Gene Set Analysis](https://mlsanity.com/paper/2509.10575/) (2025)
- [Aligning Clinical Needs and AI Capabilities: A Survey on LLMs for Medical Reasoning](https://mlsanity.com/paper/2607.07761/) (2026)
- [From Retrieval to Generation: Unifying External and Parametric Knowledge for Medical Question Answering](https://mlsanity.com/paper/2510.18297/) (2025)
- [SingProbe Technical Report](https://mlsanity.com/paper/2608.30703/) (2026)

## Abstract (arXiv)

The remarkable capabilities of Large Language Models (LLMs) make them increasingly compelling for adoption in real-world healthcare applications. However, the risks associated with using LLMs in medical applications have not been systematically characterized. We propose using five key principles for safe and trustworthy medical AI: Truthfulness, Resilience, Fairness, Robustness, and Privacy, along with ten specific aspects. Under this comprehensive framework, we introduce a novel MedGuard benchmark with 1,000 expert-verified questions. Our evaluation of 11 commonly used LLMs shows that the current language models, regardless of their safety alignment mechanisms, generally perform poorly on most of our benchmarks, particularly when compared to the high performance of human physicians. Despite recent reports indicate that advanced LLMs like ChatGPT can match or even exceed human performance in various medical tasks, this study underscores a significant safety gap, highlighting the crucial need for human oversight and the implementation of AI safety guardrails.