mlsanityData to 14 Oct 2026

MedSentry: Understanding and Mitigating Safety Risks in Medical LLM Multi-Agent Systems

arXiv 2505.20824 ↗First posted 27 May 2025cs.MA cs.AIcited by 9

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

As large language models (LLMs) are increasingly deployed in healthcare, ensuring their safety, particularly within collaborative multi-agent configurations, is paramount. In this paper we introduce MedSentry, a benchmark comprising 5 000 adversarial medical prompts spanning 25 threat categories with 100 subthemes. Coupled with this dataset, we develop an end-to-end attack-defense evaluation pipeline to systematically analyze how four representative multi-agent topologies (Layers, SharedPool, Centralized, and Decentralized) withstand attacks from 'dark-personality' agents. Our findings reveal critical differences in how these architectures handle information contamination and maintain robust decision-making, exposing their underlying vulnerability mechanisms. For instance, SharedPool's open information sharing makes it highly susceptible, whereas Decentralized architectures exhibit greater resilience thanks to inherent redundancy and isolation. To mitigate these risks, we propose a personality-scale detection and correction mechanism that identifies and rehabilitates malicious agents, restoring system safety to near-baseline levels. MedSentry thus furnishes both a rigorous evaluation framework and practical defense strategies that guide the design of safer LLM-based multi-agent systems in medical domains.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 53 of its 73 references that are in the map. First posted on arXiv 27 May 2025.

Builds on

53 of 73 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Emerging Cyber Attack Risks of Medical AI Agents2 Apr
  2. 2025Red-Teaming LLM Multi-Agent Systems via Communication Attacks20 Feb
  3. 2024Prompt Infection: LLM-to-LLM Prompt Injection within Multi-Agent Systems9 Oct
  4. 2024AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases17 Jul
  5. 2024Flooding Spread of Manipulated Knowledge in LLM-Based Multi-Agent Communities10 Jul
  6. 2024PsySafe: A Comprehensive Framework for Psychological-based Attack, Defense, and Evaluation of Multi-agent System Safety22 Jan
  7. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agents18 Jan

Other references · 46

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  3. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,808
  4. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  5. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,395
  6. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
  7. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  8. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  9. 2023ChatDev: Communicative Agents for Software Developmentcited by 919
  10. 2023Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicinecited by 329
  11. 2023Chain-of-Verification Reduces Hallucination in Large Language Modelscited by 324
  12. 2023MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoningcited by 316
34 more
  1. 2023Towards Generalist Biomedical AIcited by 289
  2. 2023MEDITRON-70B: Scaling Medical Pretraining for Large Language Modelscited by 274
  3. 2024AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environmentscited by 231
  4. 2024Agent Hospital: A Simulacrum of Hospital with Evolvable Medical Agentscited by 218
  5. 2024MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Makingcited by 163
  6. 2023Large language models in healthcare and medical domain: A reviewcited by 127
  7. 2025A Survey of LLM-based Agents in Medicine: How far are we from Baymax?cited by 85
  8. 2024AIR-Bench 2024: A Safety Benchmark Based on Risk Categories from Regulations and Policiescited by 74
  9. 2024MedSafetyBench: Evaluating and Improving the Medical Safety of Large Language Modelscited by 44
  10. 2023PoisonPrompt: Backdoor Attack on Prompt-based Large Language Modelscited by 42
  11. 2023Prompt2Model: Generating Deployable Models from Natural Language Instructionscited by 41
  12. 2024Towards Evaluating and Building Versatile Large Language Models for Medicinecited by 40
  13. 2025MedicalAgentsBench for Complex Medical Reasoning: Comparing Internalized Reasoning Models versus Externalized Agent-based Frameworkscited by 40
  14. 2025AILuminate: Introducing v1.0 of the AI Risk and Reliability Benchmark from MLCommonscited by 35
  15. 2024Polaris: A Safety-focused LLM Constellation Architecture for Healthcarecited by 28
  16. 2024MedFuzz: Exploring the Robustness of Large Language Models in Medical Question Answeringcited by 26
  17. 2025MDTeamGPT: A Self-Evolving LLM-based Multi-Agent Framework for Multi-Disciplinary Team Medical Consultationcited by 26
  18. 2025PharmAgents: Building a Virtual Pharma with Large Language Model Agentscited by 21
  19. 2023Attacking Cooperative Multi-Agent Reinforcement Learning by Adversarial Minority Influencecited by 17
  20. 2025VITAL: A New Dataset for Benchmarking Pluralistic Alignment in Healthcarecited by 17
  21. 2025Prompt Injection Detection and Mitigation via AI Multi-Agent NLP Frameworkscited by 15
  22. 2024Multi-expert Prompting Improves Reliability, Safety, and Usefulness of Large Language Modelscited by 14
  23. 2024Agentic LLM Workflows for Generating Patient-Friendly Medical Reportscited by 13
  24. 2025M3HF: Multi-agent Reinforcement Learning from Multi-phase Human Feedback of Mixed Qualitycited by 11
  25. 2024Medical MLLM is Vulnerable: Cross-Modality Jailbreak and Mismatched Attacks on Medical Multimodal Large Language Modelscited by 10
  26. 2024Adversarial Attacks on Large Language Models in Medicinecited by 10
  27. 2024Adaptive Reasoning and Acting in Medical Language Agentscited by 9
  28. 2024Prompt Injection Attacks on Large Language Models in Oncologycited by 8
  29. 2024Recent Advances in Attack and Defense Approaches of Large Language Modelscited by 6
  30. 2024RareAgents: Autonomous Multi-disciplinary Team for Rare Disease Diagnosis and Treatmentcited by 6
  31. 2024COGNET-MD, an evaluation framework and dataset for Large Language Model benchmarks in the medical domaincited by 4
  32. 2024A Proposed S.C.O.R.E. Evaluation Framework for Large Language Models : Safety, Consensus, Objectivity, Reproducibility and Explainabilitycited by 3
  33. 2024OntoMedRec: Logically-Pretrained Model-Agnostic Ontology Encoders for Medication Recommendationcited by 2
  34. 2025Hierarchical Divide-and-Conquer for Fine-Grained Alignment in LLM-Based Medical Evaluationcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 4.

9 papers.

  1. 2025A Vision for Access Control in LLM-based Agent Systemscited by 14
  2. 2025Tiered Agentic Oversight: A Hierarchical Multi-Agent System for Healthcare Safetycited by 8
  3. 2025Unified Defense for Large Language Models against Jailbreak and Fine-Tuning Attacks in Educationcited by 5
  4. 2025Chimera: Harnessing Multi-Agent LLMs for Automatic Insider Threat Simulationcited by 4
  5. 2026You Told Me to Do It: Measuring Instructional Text-induced Private Data Leakage in LLM Agentscited by 3
  6. 2025Empowering Real-World: A Survey on the Technology, Practice, and Evaluation of LLM-driven Industry Agentscited by 2
  7. 2026The Path to Self-Evolving Clinical Systems: Scaling Medical Agents from Assistance to Autonomycited by 2
  8. 2026Cooking Up Risks: Benchmarking and Reducing Food Safety Risks in Large Language Modelscited by 1
  9. 2026SoK: When Safe Agents Fail Together: The Security of Multi Agent LLM Systemscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Towards Safe AI Clinicians: A Comprehensive Study on Large Language Model Jailbreaking in Healthcare
  2. 2025Medical Malice: A Dataset for Context-Aware Safety in Healthcare LLMs
  3. 2024Ensuring Safety and Trust: Analyzing the Risks of Large Language Models in Medicine
  4. 2025CARES: Comprehensive Evaluation of Safety and Adversarial Robustness in Medical LLMs
  5. 2026Improving the Safety and Trustworthiness of Medical AI via Multi-Agent Evaluation Loops
  6. 2026Toward Reliable, Safe, and Secure LLMs for Scientific Applications
  7. 2025Balancing Safety and Helpfulness in Healthcare AI Assistants through Iterative Preference Alignment
  8. 2026A Multi-Domain Red Teaming Framework for Safety, Robustness, and Fairness Evaluation of Medical Large Language Models