mlsanityData to 14 Oct 2026

Red Teaming Large Language Models for Healthcare

arXiv 2505.00467 ↗First posted 1 May 2025cs.CL cs.AIcited by 2

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

We present the design process and findings of the pre-conference workshop at the Machine Learning for Healthcare Conference (2024) entitled Red Teaming Large Language Models for Healthcare, which took place on August 15, 2024. Conference participants, comprising a mix of computational and clinical expertise, attempted to discover vulnerabilities -- realistic clinical prompts for which a large language model (LLM) outputs a response that could cause clinical harm. Red-teaming with clinicians enables the identification of LLM vulnerabilities that may not be recognised by LLM developers lacking clinical expertise. We report the vulnerabilities found, categorise them, and present the results of a replication study assessing the vulnerabilities across all LLMs provided.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 7 of its 24 references that are in the map. First posted on arXiv 1 May 2025.

Builds on

7 of 24 references.

References · 7

  1. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  2. 2024Red-Teaming for Generative AI: Silver Bullet or Security Theater?cited by 83
  3. 2024Against The Achilles' Heel: A Survey on Red Teaming for Generative Modelscited by 59
  4. 2023Evaluation of GPT-3.5 and GPT-4 for supporting real-world information needs in healthcare deliverycited by 30
  5. 2025Lessons From Red Teaming 100 Generative AI Productscited by 23
  6. 2023Walking a Tightrope -- Evaluating Large Language Models in High-Risk Domainscited by 15
  7. 2024Beyond Accuracy: Investigating Error Types in GPT-4 Responses to USMLE Questionscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026ResidencyRL: Reinforcement Learning in Simulated Clinical Environmentscited by 3
  2. 2026A Multi-Domain Red Teaming Framework for Safety, Robustness, and Fairness Evaluation of Medical Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Addressing Benchmarking Gaps in Large Language Models for Health and Medicine with Dynamic Red-Teaming
  2. 2026JMedEthicBench: A Multi-Turn Conversational Benchmark for Evaluating Medical Safety in Japanese Large Language Models
  3. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignment
  4. 2024Ensuring Safety and Trust: Analyzing the Risks of Large Language Models in Medicine
  5. 2025Large language models provide unsafe answers to patient-posed medical questions
  6. 2025Aligning Large Language Models with Healthcare Stakeholders: A Pathway to Trustworthy AI Integration
  7. 2026When Large Language Models Fail in Healthcare: Evaluating Sensitivity to Prompt Variations
  8. 2026A primer on evaluation methods for large language models in healthcare