mlsanityData to 14 Oct 2026

MedSafetyBench: Evaluating and Improving the Medical Safety of Large Language Models

arXiv 2403.03744 ↗First posted 6 Mar 2024cs.AIcited by 44

Not on the map. Why?

Abstract (arXiv)

As large language models (LLMs) develop increasingly sophisticated capabilities and find applications in medical settings, it becomes important to assess their medical safety due to their far-reaching implications for personal and public health, patient safety, and human rights. However, there is little to no understanding of the notion of medical safety in the context of LLMs, let alone how to evaluate and improve it. To address this gap, we first define the notion of medical safety in LLMs based on the Principles of Medical Ethics set forth by the American Medical Association. We then leverage this understanding to introduce MedSafetyBench, the first benchmark dataset designed to measure the medical safety of LLMs. We demonstrate the utility of MedSafetyBench by using it to evaluate and improve the medical safety of LLMs. Our results show that publicly-available medical LLMs do not meet standards of medical safety and that fine-tuning them using MedSafetyBench improves their medical safety while preserving their medical performance. By introducing this new benchmark dataset, our work enables a systematic study of the state of medical safety in LLMs and motivates future work in this area, paving the way to mitigate the safety risks of LLMs in medicine. The benchmark dataset and code are available at https://github.com/AI4LIFE-GROUP/med-safety-bench.

Key facts. Cited by 44 papers in the map (data through 14 Oct 2026). Builds on 29 of its 48 references that are in the map. First posted on arXiv 6 Mar 2024.

Builds on

29 of 48 references.

References · 29

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2017Deep reinforcement learning from human preferencescited by 3,912
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  10. 2023Mistral 7Bcited by 3,164
  11. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  12. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
17 more
  1. 2024Mixtral of Expertscited by 1,823
  2. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  3. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
  4. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  5. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  6. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  7. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  8. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 543
  9. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 387
  10. 2023MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Datacited by 286
  11. 2023MEDITRON-70B: Scaling Medical Pretraining for Large Language Modelscited by 274
  12. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignmentcited by 223
  13. 2023SafetyBench: Evaluating the Safety of Large Language Modelscited by 199
  14. 2021Alignment of Language Agentscited by 168
  15. 2023PMC-LLaMA: Towards Building Open-source Language Models for Medicinecited by 88
  16. 2021Consequences of Misaligned AIcited by 80
  17. 2024Towards Conversational Diagnostic AIcited by 79

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 23; 2026: 18.

44 papers.

Sort:

By area:

  1. 2025CARES: Comprehensive Evaluation of Safety and Adversarial Robustness in Medical LLMscited by 26
  2. 2025The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMscited by 26
  3. 2024Leveraging Large Language Model as Simulated Patients for Clinical Educationcited by 21
  4. 2025Large language models provide unsafe answers to patient-posed medical questionscited by 18
  5. 2024A Survey on Large Language Models from General Purpose to Medical Applications: Datasets, Methodologies, and Evaluationscited by 16
  6. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 15
  7. 2025Towards Safe AI Clinicians: A Comprehensive Study on Large Language Model Jailbreaking in Healthcarecited by 10
  8. 2025MedSentry: Understanding and Mitigating Safety Risks in Medical LLM Multi-Agent Systemscited by 9
  9. 2025CLINIC: Evaluating Multilingual Trustworthiness in Language Models for Healthcarecited by 9
  10. 2024From Text to Multimodality: Exploring the Evolution and Impact of Large Language Models in Medical Practicecited by 8
  11. 2025Tiered Agentic Oversight: A Hierarchical Multi-Agent System for Healthcare Safetycited by 8
  12. 2025TRIDENT: Benchmarking LLM Safety in Finance, Medicine, and Lawcited by 8
32 more
  1. 2025When Style Breaks Safety: Defending LLMs Against Superficial Style Alignmentcited by 6
  2. 2025Unified Defense for Large Language Models against Jailbreak and Fine-Tuning Attacks in Educationcited by 5
  3. 2025NeuronTune: Fine-Grained Neuron Modulation for Balanced Safety-Utility Alignment in LLMscited by 4
  4. 2025ProGuard: Towards Proactive Multimodal Safeguardcited by 4
  5. 2025Grounded in Reality: Learning and Deploying Proactive LLM from Offline Logscited by 3
  6. 2026RedBench: A Universal Dataset for Comprehensive Red Teaming of Large Language Modelscited by 3
  7. 2026Improving the Safety and Trustworthiness of Medical AI via Multi-Agent Evaluation Loopscited by 3
  8. 2026Using LLM-as-a-Judge/Jury to Advance Scalable, Clinically-Validated Safety Evaluations of Model Responses to Users Demonstrating Psychosiscited by 3
  9. 2025VocalAgent: Large Language Models for Vocal Health Diagnostics with Safety-Aware Evaluationcited by 2
  10. 2025PAM: Training Policy-Aligned Moderation Filters at Scalecited by 2
  11. 2026The Path to Self-Evolving Clinical Systems: Scaling Medical Agents from Assistance to Autonomycited by 2
  12. 2026Layered LLM Defenses as an Ensemble: Access Tiers, Inference Cost, and the Measured Failure Correlation Between Defense Layerscited by 2
  13. 2025Automatic Replication of LLM Mistakes in Medical Conversationscited by 1
  14. 2026Benchmarking the Safety of Large Language Models for Robotic Health Attendant Controlcited by 1
  15. 2026ALIGNBEAM : Inference-Time Alignment Transfer via Cross-Vocabulary Logit Mixingcited by 1
  16. 2025NurValues: Real-World Nursing Values Evaluation for Large Language Models in Clinical Contextcited by 0
  17. 2025Red Teaming for Generative AI, Report on a Copyright-Focused Exercise Completed in an Academic Medical Centercited by 0
  18. 2025Text2VLM: Adapting Text-Only Datasets to Evaluate Alignment Training in Visual Language Modelscited by 0
  19. 2025Mirror, Mirror on the Wall -- Which is the Best Model of Them All?cited by 0
  20. 2025A Practical Framework for Evaluating Medical AI Security: Reproducible Assessment of Jailbreaking and Privacy Vulnerabilities Across Clinical Specialtiescited by 0
  21. 2025The Forgotten Shield: Safety Grafting in Parameter-Space for Medical MLLMscited by 0
  22. 2026Deliberative multi-agent large language models improve clinical reasoning in ophthalmologycited by 0
  23. 2026Towards Understanding the Robustness of Sparse Autoencoderscited by 0
  24. 2026CareGuardAI: Context-Aware Multi-Agent Guardrails for Clinical Safety & Hallucination Mitigation in Patient-Facing LLMscited by 0
  25. 2026A Multi-Domain Red Teaming Framework for Safety, Robustness, and Fairness Evaluation of Medical Large Language Modelscited by 0
  26. 2026Dialectics of Alignment: Harnessing Unsafe Knowledge for Dynamic Safety Routingcited by 0
  27. 2026DrugBench: Evaluating AI Control Protocols for Medication Harm Mitigationcited by 0
  28. 2026Trustworthiness Costs of Domain Adaptation in Small Language Models:A Cross-Architecture Empirical Studycited by 0
  29. 2026A Reproducible, License-Aware Distillation Recipe for CPUDeployable Safety Classificationcited by 0
  30. 2026Beyond Accuracy: Counterfactual Fragility and Demographic Bias in Clinical Evaluation of LLMscited by 0
  31. 2026Applying Language Models in Clinical Medicine: Recent Trends and Perspectivescited by 0
  32. 2026Generalization Is Stability, Not Accuracy: Multi-Axis Evaluation of LLMscited by 0