mlsanityData to 7 Oct 2026

CARES: Comprehensive Evaluation of Safety and Adversarial Robustness in Medical LLMs

arXiv 2505.11413 ↗First posted 16 May 2025cs.CLcited by 26

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

Large language models (LLMs) are increasingly deployed in medical contexts, raising critical concerns about safety, alignment, and susceptibility to adversarial manipulation. While prior benchmarks assess model refusal capabilities for harmful prompts, they often lack clinical specificity, graded harmfulness levels, and coverage of jailbreak-style attacks. We introduce CARES (Clinical Adversarial Robustness and Evaluation of Safety), a benchmark for evaluating LLM safety in healthcare. CARES includes over 18,000 prompts spanning eight medical safety principles, four harm levels, and four prompting styles: direct, indirect, obfuscated, and role-play, to simulate both malicious and benign use cases. We propose a three-way response evaluation protocol (Accept, Caution, Refuse) and a fine-grained Safety Score metric to assess model behavior. Our analysis reveals that many state-of-the-art LLMs remain vulnerable to jailbreaks that subtly rephrase harmful prompts, while also over-refusing safe but atypically phrased queries. Finally, we propose a mitigation strategy using a lightweight classifier to detect jailbreak attempts and steer models toward safer behavior via reminder-based conditioning. CARES provides a rigorous framework for testing and improving medical LLM safety under adversarial and ambiguous conditions.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 32 of its 55 references that are in the map. First posted on arXiv 16 May 2025.

Builds on

32 of 55 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024SG-Bench: Evaluating LLM Safety Generalization Across Diverse Tasks and Prompt Types29 Oct
  2. 2024Role-Play Paradox in Large Language Models: Reasoning Performance Gains and Ethical Dilemmas21 Sep
  3. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preference20 Jun

Other references · 29

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024Qwen2.5 Technical Reportcited by 4,713
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  4. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  6. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  7. 2024Mixtral of Expertscited by 1,785
  8. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  9. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  10. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  11. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 630
  12. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 478
17 more
  1. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 419
  2. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 295
  3. 2017Generalized End-to-End Loss for Speaker Verificationcited by 290
  4. 2023MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Datacited by 282
  5. 2024Agent-SafetyBench: Evaluating the Safety of LLM Agentscited by 280
  6. 2023MEDITRON-70B: Scaling Medical Pretraining for Large Language Modelscited by 272
  7. 2024Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalizationcited by 203
  8. 2023SafetyBench: Evaluating the Safety of Large Language Modelscited by 189
  9. 2024Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attackscited by 150
  10. 2024Benchmarking Complex Instruction-Following with Multiple Constraints Compositioncited by 147
  11. 2018The Adverse Effects of Code Duplication in Machine Learning Models of Codecited by 123
  12. 2024Collective Constitutional AI: Aligning a Language Model with Public Inputcited by 120
  13. 2024GUARD: Role-playing to Generate Natural-language Jailbreakings to Test Guideline Adherence of Large Language Modelscited by 44
  14. 2024MedSafetyBench: Evaluating and Improving the Medical Safety of Large Language Modelscited by 43
  15. 2024Measuring and Benchmarking Large Language Models' Capabilities to Generate Persuasive Languagecited by 31
  16. 2025Shh, don't say that! Domain Certification in LLMscited by 7
  17. 2025Data-adaptive Safety Rules for Training Reward Modelscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 19.

26 papers.

Sort:

By area:

  1. 2025From ASR to ASP: Evaluating Prompt Attack Vulnerabilities Against Open-Source LLMscited by 6
  2. 2026SycoEval-EM: Sycophancy Evaluation of Large Language Models in Simulated Clinical Encounters for Emergency Carecited by 5
  3. 2026MatrAIx: Simulating the World with 8.3 Billion Persona Agentscited by 5
  4. 2025Mitigating Over-Refusal in Aligned Large Language Models via Inference-Time Activation Energycited by 4
  5. 2026Health-ORSC-Bench: A Benchmark for Measuring Over-Refusal and Safety Completion in Health Contextcited by 4
  6. 2026Do No Harm: Exposing Hidden Vulnerabilities of LLMs via Persona-based Client Simulation Attack in Psychological Counselingcited by 4
  7. 2026Faithfulness vs. Safety: Evaluating LLM Behavior Under Counterfactual Medical Evidencecited by 3
  8. 2026MPIB: A Benchmark for Medical Prompt Injection Attacks and Clinical Safety in LLMscited by 3
  9. 2025mFARM: Towards Multi-Faceted Fairness Assessment based on HARMs in Clinical Decision Supportcited by 2
  10. 2026LegalCiteBench: Evaluating Citation Reliability in Legal Language Modelscited by 2
  11. 2025Analysing Environmental Efficiency in AI for X-Ray Diagnosiscited by 1
  12. 2025Balancing Safety and Helpfulness in Healthcare AI Assistants through Iterative Preference Alignmentcited by 1
14 more
  1. 2026Chain of Risk: Safety Failures in Large Reasoning Models and Mitigation via Adaptive Multi-Principle Steeringcited by 1
  2. 2026Aligning Clinical Needs and AI Capabilities: A Survey on LLMs for Medical Reasoningcited by 1
  3. 2025The Forgotten Shield: Safety Grafting in Parameter-Space for Medical MLLMscited by 0
  4. 2025Safety boundary maintenance in consumer AI systems responding to pediatric health queries: a cross-platform benchmark evaluation under naturalistic and adversarially pressured conditionscited by 0
  5. 2026MedRedFlag: Investigating how LLMs Redirect Misconceptions in Real-World Health Communicationcited by 0
  6. 2026Do Web Agents Investigate Before They Decide?cited by 0
  7. 2026How Robust Are Large Language Models for Clinical Numeracy? An Empirical Study on Numerical Reasoning Abilities in Clinical Contextscited by 0
  8. 2026Understanding Annotator Safety Policy with Interpretabilitycited by 0
  9. 2026A Security Analysis of Long-Horizon Agentic AI Systems: Threats, Evaluation, and Framework Developmentcited by 0
  10. 2026PolicyAlign: Direct Policy-Based Safety Alignment for Large Language Modelscited by 0
  11. 2026When Medical Safety Alignment Fails: A Benchmark for Evaluating LLMs on High-Risk Medical Queriescited by 0
  12. 2026Coupled Graph--Policy Distillation for Personalized Medication Safety in Older Adults with Multimorbiditycited by 0
  13. 2026Sycophants in the Courtroom: Are LLMs Fragile to Juridical Authority and Evolving Legal Standards?cited by 0
  14. 2026EmoMed: An Emotionally-Aware Agent for Multimodal Medical Support with Real-Time Information Retrievalcited by 0