mlsanityData to 7 Oct 2026

A Practical Framework for Evaluating Medical AI Security: Reproducible Assessment of Jailbreaking and Privacy Vulnerabilities Across Clinical Specialties

arXiv 2512.08185 ↗First posted 9 Dec 2025cs.CR cs.AIcited by 0

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

Medical Large Language Models (LLMs) are increasingly deployed for clinical decision support across diverse specialties, yet systematic evaluation of their robustness to adversarial misuse and privacy leakage remains inaccessible to most researchers. Existing security benchmarks require GPU clusters, commercial API access, or protected health data -- barriers that limit community participation in this critical research area. We propose a practical, fully reproducible framework for evaluating medical AI security under realistic resource constraints. Our framework design covers multiple medical specialties stratified by clinical risk -- from high-risk domains such as emergency medicine and psychiatry to general practice -- addressing jailbreaking attacks (role-playing, authority impersonation, multi-turn manipulation) and privacy extraction attacks. All evaluation utilizes synthetic patient records requiring no IRB approval. The framework is designed to run entirely on consumer CPU hardware using freely available models, eliminating cost barriers. We present the framework specification including threat models, data generation methodology, evaluation protocols, and scoring rubrics. This proposal establishes a foundation for comparative security assessment of medical-specialist models and defense mechanisms, advancing the broader goal of ensuring safe and trustworthy medical AI systems.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 32 of its 42 references that are in the map. First posted on arXiv 9 Dec 2025.

Builds on

32 of 42 references.

References · 32

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  6. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  7. 2020Extracting Training Data from Large Language Modelscited by 2,144
  8. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  9. 2016Concrete Problems in AI Safetycited by 1,582
  10. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  11. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  12. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
20 more
  1. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  2. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  3. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  4. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 738
  5. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  6. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
  7. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  8. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 531
  9. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 525
  10. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 505
  11. 2019ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmissioncited by 487
  12. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 434
  13. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 415
  14. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 387
  15. 2023Towards Expert-Level Medical Question Answering with Large Language Modelscited by 382
  16. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  17. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Modelscited by 307
  18. 2023MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbotscited by 172
  19. 2024MedSafetyBench: Evaluating and Improving the Medical Safety of Large Language Modelscited by 43
  20. 2021Domain-Specific Pretraining for Vertical Search: Case Study on Biomedical Literaturecited by 6

Built on by

Not cited yet.