mlsanityData to 14 Oct 2026

Medical Malice: A Dataset for Context-Aware Safety in Healthcare LLMs

arXiv 2511.21757 ↗First posted 24 Nov 2025cs.CY cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

The integration of Large Language Models (LLMs) into healthcare demands a safety paradigm rooted in textitprimum non nocere. However, current alignment techniques rely on generic definitions of harm that fail to capture context-dependent violations, such as administrative fraud and clinical discrimination. To address this, we introduce Medical Malice: a dataset of 214,219 adversarial prompts calibrated to the regulatory and ethical complexities of the Brazilian Unified Health System (SUS). Crucially, the dataset includes the reasoning behind each violation, enabling models to internalize ethical boundaries rather than merely memorizing a fixed set of refusals. Using an unaligned agent (Grok-4) within a persona-driven pipeline, we synthesized high-fidelity threats across seven taxonomies, ranging from procurement manipulation and queue-jumping to obstetric violence. We discuss the ethical design of releasing these "vulnerability signatures" to correct the information asymmetry between malicious actors and AI developers. Ultimately, this work advocates for a shift from universal to context-aware safety, providing the necessary resources to immunize healthcare AI against the nuanced, systemic threats inherent to high-stakes medical environments -- vulnerabilities that represent the paramount risk to patient safety and the successful integration of AI in healthcare systems.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 5 of its 17 references that are in the map. First posted on arXiv 24 Nov 2025.

Builds on

5 of 17 references.

References · 5

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  3. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  4. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  5. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 638

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025CARES: Comprehensive Evaluation of Safety and Adversarial Robustness in Medical LLMs
  2. 2024Ensuring Safety and Trust: Analyzing the Risks of Large Language Models in Medicine
  3. 2026Improving the Safety and Trustworthiness of Medical AI via Multi-Agent Evaluation Loops
  4. 2025Towards Safe AI Clinicians: A Comprehensive Study on Large Language Model Jailbreaking in Healthcare
  5. 2025MedSentry: Understanding and Mitigating Safety Risks in Medical LLM Multi-Agent Systems
  6. 2025Balancing Safety and Helpfulness in Healthcare AI Assistants through Iterative Preference Alignment
  7. 2026Health-ORSC-Bench: A Benchmark for Measuring Over-Refusal and Safety Completion in Health Context
  8. 2025A Practical Framework for Evaluating Medical AI Security: Reproducible Assessment of Jailbreaking and Privacy Vulnerabilities Across Clinical Specialties