mlsanityData to 7 Oct 2026

Towards Safe AI Clinicians: A Comprehensive Study on Large Language Model Jailbreaking in Healthcare

arXiv 2501.18632 ↗First posted 27 Jan 2025cs.CR cs.CLcited by 10

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

Large language models (LLMs) are increasingly utilized in healthcare applications. However, their deployment in clinical practice raises significant safety concerns, including the potential spread of harmful information. This study systematically assesses the vulnerabilities of seven LLMs to three advanced black-box jailbreaking techniques within medical contexts. To quantify the effectiveness of these techniques, we propose an automated and domain-adapted agentic evaluation pipeline. Experiment results indicate that leading commercial and open-source LLMs are highly vulnerable to medical jailbreaking attacks. To bolster model safety and reliability, we further investigate the effectiveness of Continual Fine-Tuning (CFT) in defending against medical adversarial attacks. Our findings underscore the necessity for evolving attack methods evaluation, domain-specific safety alignment, and LLM safety-utility balancing. This research offers actionable insights for advancing the safety and reliability of AI clinicians, contributing to ethical and effective AI deployment in healthcare.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 24 of its 30 references that are in the map. First posted on arXiv 27 Jan 2025.

Builds on

24 of 30 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct

Other references · 23

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  6. 2024GPT-4o System Cardcited by 4,467
  7. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  8. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  9. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  10. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 533
  11. 2024A StrongREJECT for Empty Jailbreakscited by 352
  12. 2023MEDITRON-70B: Scaling Medical Pretraining for Large Language Modelscited by 272
11 more
  1. 2023A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easilycited by 242
  2. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 215
  3. 2023Med-HALT: Medical Domain Hallucination Test for Large Language Modelscited by 156
  4. 2024FlipAttack: Jailbreak LLMs via Flippingcited by 73
  5. 2024CodeChameleon: Personalized Encryption Framework for Jailbreaking Large Language Modelscited by 65
  6. 2024A Toolbox for Surfacing Health Equity Harms and Biases in Large Language Modelscited by 45
  7. 2024The Ethics of ChatGPT in Medicine and Healthcare: A Systematic Review on Large Language Models (LLMs)cited by 45
  8. 2024MedSafetyBench: Evaluating and Improving the Medical Safety of Large Language Modelscited by 43
  9. 2024JailbreakEval: An Integrated Toolkit for Evaluating Jailbreak Attempts Against Large Language Modelscited by 28
  10. 2024Mitigating the Risk of Health Inequity Exacerbated by Large Language Modelscited by 21
  11. 2024An Interpretable N-gram Perplexity Threat Model for Large Language Model Jailbreakscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 7.

10 papers.

  1. 2026AgentLAB: Benchmarking LLM Agents against Long-Horizon Attackscited by 31
  2. 2025Factuality Beyond Coherence: Evaluating LLM Watermarking Methods for Medical Textscited by 4
  3. 2026RASA: Routing-Aware Safety Alignment for Mixture-of-Experts Modelscited by 3
  4. 2026Beyond Idealized Patients: Evaluating LLMs under Challenging Patient Behaviors in Medical Consultationscited by 3
  5. 2026Scaling Medical Reasoning Verification via Tool-Integrated Reinforcement Learningcited by 2
  6. 2025An Audit and Analysis of LLM-Assisted Health Misinformation Jailbreaks Against LLMscited by 1
  7. 2025Balancing Safety and Helpfulness in Healthcare AI Assistants through Iterative Preference Alignmentcited by 1
  8. 2026Toward Reliable, Safe, and Secure LLMs for Scientific Applicationscited by 1
  9. 2026MedRedFlag: Investigating how LLMs Redirect Misconceptions in Real-World Health Communicationcited by 0
  10. 2026One Year Later...The Harms Persist, But So Do We!cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Ensuring Safety and Trust: Analyzing the Risks of Large Language Models in Medicine
  2. 2026Improving the Safety and Trustworthiness of Medical AI via Multi-Agent Evaluation Loops
  3. 2025MedSentry: Understanding and Mitigating Safety Risks in Medical LLM Multi-Agent Systems
  4. 2024Large Language Model Safety: A Holistic Survey
  5. 2025The Forgotten Shield: Safety Grafting in Parameter-Space for Medical MLLMs
  6. 2025Many-to-One Adversarial Consensus: Exposing Multi-Agent Collusion Risks in AI-Based Healthcare
  7. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks
  8. 2024Medical MLLM is Vulnerable: Cross-Modality Jailbreak and Mismatched Attacks on Medical Multimodal Large Language Models