Trivial Prompt Reframing Bypasses Safety Guardrails in Google's MedGemma-4B
On the map: LLM safety and security → Safety alignment, refusal and guardrails
Abstract (arXiv)
Open-weight medical language models are increasingly used as the base of patient-facing and clinician-support applications. Their model cards prohibit specific behaviors -- recommending exact drug dosages, issuing definitive diagnoses, prescribing treatments, adjudicating drug-drug interactions, and advising that emergency care can be skipped -- yet a model card describes intended behavior, not robust behavior. We quantify that gap for MedGemma-4B-it under attacks that require no technical sophistication. We build a fully factorial benchmark of 5 guarded-behavior concepts x 50 deterministically templated questions x 6 lay-accessible attack manners x 3 repetitions (4,500 generations), serve the model locally through Ollama under default sampling, and code every response refuse/hedge/comply with three independent judges (an LLM judge, a transparent regex judge, and an NLI-entailment judge). Under the primary LLM judge the overall Attack Success Rate (ASR, the fraction coded comply) is 38.0%. The two framings that reinterpret the request as legitimate dominate: recasting a question as a "medical board exam" item raises ASR from a 29.0% baseline to 53.1% (+24.0 points), and an appeal to an alleged doctor's authority raises it to 43.7% (+14.7); crude instruction-override prefixes have no significant effect. Robustness is dominated by topic: the drug-interaction guardrail is nearly absent (83.2% ASR) while the emergency-deferral guardrail is strong (4.7%) -- and the authority framing is the only attack that breaches it. We report Wilson confidence intervals, cluster-bootstrap effect sizes, a cluster-robust logistic regression, Cochran's Q, per-manner McNemar tests, and inter-judge reliability (Fleiss' kappa = 0.26); absolute ASR is judge-dependent while the ordering of attacks and topics is not. Our findings motivate stronger deployment-time guardrails for open medical models.
Key facts. No citing papers in the map yet (data through 13 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 10 of its 20 references that are in the map. First posted on arXiv 9 Jul 2026.
Builds on
10 of 20 references.
References · 10
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,492
- 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,064
- 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,676
- 2022Large Language Models Encode Clinical Knowledgecited by 1,817
- 2025Gemma 3 Technical Reportcited by 1,748
- 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,637
- 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,417
- 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,404
- 2023Towards Understanding Sycophancy in Language Modelscited by 1,018
- 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 817
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025CARES: Comprehensive Evaluation of Safety and Adversarial Robustness in Medical LLMs
- 2026When Chain-of-Thought Backfires: Evaluating Prompt Sensitivity in Medical Language Models
- 2025Towards Safe AI Clinicians: A Comprehensive Study on Large Language Model Jailbreaking in Healthcare
- 2025MedOmni-45deg: A Safety-Performance Benchmark for Reasoning-Oriented LLMs in Medicine
- 2026MindGuard: Guardrail Classifiers for Multi-Turn Mental Health Support
- 2026Health-ORSC-Bench: A Benchmark for Measuring Over-Refusal and Safety Completion in Health Context
- 2024Ensuring Safety and Trust: Analyzing the Risks of Large Language Models in Medicine
- 2024Medical MLLM is Vulnerable: Cross-Modality Jailbreak and Mismatched Attacks on Medical Multimodal Large Language Models