# CARES: Comprehensive Evaluation of Safety and Adversarial Robustness in Medical LLMs

arXiv: https://arxiv.org/abs/2505.11413 · HTML: https://mlsanity.com/paper/2505.11413/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security.
- Builds on 32 of its 55 references that are in the map.
- First posted on arXiv 16 May 2025.

## Placed in

- LLM safety and security / [Safety alignment, refusal and guardrails](https://mlsanity.com/safety/safety-alignment-refusal-and-guardrails/)

## Builds on (most cited first, 30 of 32)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [Jailbroken: How Does LLM Safety Training Fail?](https://mlsanity.com/paper/2307.02483/) (2023)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [Deduplicating Training Data Makes Language Models Better](https://mlsanity.com/paper/2107.06499/) (2021)
- [Capabilities of GPT-4 on Medical Challenge Problems](https://mlsanity.com/paper/2303.13375/) (2023)
- [Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study](https://mlsanity.com/paper/2305.13860/) (2023)
- [ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection](https://mlsanity.com/paper/2203.09509/) (2022)
- [Agent-SafetyBench: Evaluating the Safety of LLM Agents](https://mlsanity.com/paper/2412.14470/) (2024)
- [SemDeDup: Data-efficient learning at web-scale through semantic deduplication](https://mlsanity.com/paper/2303.09540/) (2023)
- [Generalized End-to-End Loss for Speaker Verification](https://mlsanity.com/paper/1710.10467/) (2017)
- [MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Data](https://mlsanity.com/paper/2304.08247/) (2023)
- [MEDITRON-70B: Scaling Medical Pretraining for Large Language Models](https://mlsanity.com/paper/2311.16079/) (2023)
- [PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preference](https://mlsanity.com/paper/2406.15513/) (2024)
- [Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalization](https://mlsanity.com/paper/2406.01171/) (2024)
- [SafetyBench: Evaluating the Safety of Large Language Models](https://mlsanity.com/paper/2309.07045/) (2023)
- [Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attacks](https://mlsanity.com/paper/2401.17263/) (2024)
- [Benchmarking Complex Instruction-Following with Multiple Constraints Composition](https://mlsanity.com/paper/2407.03978/) (2024)
- [The Adverse Effects of Code Duplication in Machine Learning Models of Code](https://mlsanity.com/paper/1812.06469/) (2018)
- [Collective Constitutional AI: Aligning a Language Model with Public Input](https://mlsanity.com/paper/2406.07814/) (2024)
- [SG-Bench: Evaluating LLM Safety Generalization Across Diverse Tasks and Prompt Types](https://mlsanity.com/paper/2410.21965/) (2024)
- [GUARD: Role-playing to Generate Natural-language Jailbreakings to Test Guideline Adherence of Large Language Models](https://mlsanity.com/paper/2402.03299/) (2024)
- [MedSafetyBench: Evaluating and Improving the Medical Safety of Large Language Models](https://mlsanity.com/paper/2403.03744/) (2024)
- [Measuring and Benchmarking Large Language Models' Capabilities to Generate Persuasive Language](https://mlsanity.com/paper/2406.17753/) (2024)
- [Shh, don't say that! Domain Certification in LLMs](https://mlsanity.com/paper/2502.19320/) (2025)

## Built on by (most cited first, 26 of 26)

- [MatrAIx: Simulating the World with 8.3 Billion Persona Agents](https://mlsanity.com/paper/2608.04205/) (2026)
- [From ASR to ASP: Evaluating Prompt Attack Vulnerabilities Against Open-Source LLMs](https://mlsanity.com/paper/2505.14368/) (2025)
- [Mitigating Over-Refusal in Aligned Large Language Models via Inference-Time Activation Energy](https://mlsanity.com/paper/2510.08646/) (2025)
- [SycoEval-EM: Sycophancy Evaluation of Large Language Models in Simulated Clinical Encounters for Emergency Care](https://mlsanity.com/paper/2601.16529/) (2026)
- [Health-ORSC-Bench: A Benchmark for Measuring Over-Refusal and Safety Completion in Health Context](https://mlsanity.com/paper/2601.17642/) (2026)
- [Do No Harm: Exposing Hidden Vulnerabilities of LLMs via Persona-based Client Simulation Attack in Psychological Counseling](https://mlsanity.com/paper/2604.04842/) (2026)
- [Faithfulness vs. Safety: Evaluating LLM Behavior Under Counterfactual Medical Evidence](https://mlsanity.com/paper/2601.11886/) (2026)
- [MPIB: A Benchmark for Medical Prompt Injection Attacks and Clinical Safety in LLMs](https://mlsanity.com/paper/2602.06268/) (2026)
- [LegalCiteBench: Evaluating Citation Reliability in Legal Language Models](https://mlsanity.com/paper/2605.10186/) (2026)
- [mFARM: Towards Multi-Faceted Fairness Assessment based on HARMs in Clinical Decision Support](https://mlsanity.com/paper/2509.02007/) (2025)
- [Analysing Environmental Efficiency in AI for X-Ray Diagnosis](https://mlsanity.com/paper/2511.07436/) (2025)
- [Balancing Safety and Helpfulness in Healthcare AI Assistants through Iterative Preference Alignment](https://mlsanity.com/paper/2512.04210/) (2025)
- [Chain of Risk: Safety Failures in Large Reasoning Models and Mitigation via Adaptive Multi-Principle Steering](https://mlsanity.com/paper/2605.05678/) (2026)
- [PolicyAlign: Direct Policy-Based Safety Alignment for Large Language Models](https://mlsanity.com/paper/2606.25442/) (2026)
- [Aligning Clinical Needs and AI Capabilities: A Survey on LLMs for Medical Reasoning](https://mlsanity.com/paper/2607.07761/) (2026)
- [The Forgotten Shield: Safety Grafting in Parameter-Space for Medical MLLMs](https://mlsanity.com/paper/2601.04199/) (2025)
- [Safety boundary maintenance in consumer AI systems responding to pediatric health queries: a cross-platform benchmark evaluation under naturalistic and adversarially pressured conditions](https://mlsanity.com/paper/2601.09721/) (2025)
- [MedRedFlag: Investigating how LLMs Redirect Misconceptions in Real-World Health Communication](https://mlsanity.com/paper/2601.09853/) (2026)
- [Do Web Agents Investigate Before They Decide?](https://mlsanity.com/paper/2602.05354/) (2026)
- [How Robust Are Large Language Models for Clinical Numeracy? An Empirical Study on Numerical Reasoning Abilities in Clinical Contexts](https://mlsanity.com/paper/2604.11133/) (2026)
- [Understanding Annotator Safety Policy with Interpretability](https://mlsanity.com/paper/2605.05329/) (2026)
- [A Security Analysis of Long-Horizon Agentic AI Systems: Threats, Evaluation, and Framework Development](https://mlsanity.com/paper/2606.14816/) (2026)
- [When Medical Safety Alignment Fails: A Benchmark for Evaluating LLMs on High-Risk Medical Queries](https://mlsanity.com/paper/2606.28332/) (2026)
- [Coupled Graph--Policy Distillation for Personalized Medication Safety in Older Adults with Multimorbidity](https://mlsanity.com/paper/2608.09443/) (2026)
- [Sycophants in the Courtroom: Are LLMs Fragile to Juridical Authority and Evolving Legal Standards?](https://mlsanity.com/paper/2608.21409/) (2026)
- [EmoMed: An Emotionally-Aware Agent for Multimodal Medical Support with Real-Time Information Retrieval](https://mlsanity.com/paper/2609.07194/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are increasingly deployed in medical contexts, raising critical concerns about safety, alignment, and susceptibility to adversarial manipulation. While prior benchmarks assess model refusal capabilities for harmful prompts, they often lack clinical specificity, graded harmfulness levels, and coverage of jailbreak-style attacks. We introduce CARES (Clinical Adversarial Robustness and Evaluation of Safety), a benchmark for evaluating LLM safety in healthcare. CARES includes over 18,000 prompts spanning eight medical safety principles, four harm levels, and four prompting styles: direct, indirect, obfuscated, and role-play, to simulate both malicious and benign use cases. We propose a three-way response evaluation protocol (Accept, Caution, Refuse) and a fine-grained Safety Score metric to assess model behavior. Our analysis reveals that many state-of-the-art LLMs remain vulnerable to jailbreaks that subtly rephrase harmful prompts, while also over-refusing safe but atypically phrased queries. Finally, we propose a mitigation strategy using a lightweight classifier to detect jailbreak attempts and steer models toward safer behavior via reminder-based conditioning. CARES provides a rigorous framework for testing and improving medical LLM safety under adversarial and ambiguous conditions.