mlsanityData to 14 Oct 2026

Emerging Cyber Attack Risks of Medical AI Agents

arXiv 2504.03759 ↗First posted 2 Apr 2025cs.CR cs.AIcited by 8

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

Large language models (LLMs)-powered AI agents exhibit a high level of autonomy in addressing medical and healthcare challenges. With the ability to access various tools, they can operate within an open-ended action space. However, with the increase in autonomy and ability, unforeseen risks also arise. In this work, we investigated one particular risk, i.e., cyber attack vulnerability of medical AI agents, as agents have access to the Internet through web browsing tools. We revealed that through adversarial prompts embedded on webpages, cyberattackers can: i) inject false information into the agent's response; ii) they can force the agent to manipulate recommendation (e.g., healthcare products and services); iii) the attacker can also steal historical conversations between the user and agent, resulting in the leak of sensitive/private medical information; iv) furthermore, the targeted agent can also cause a computer system hijack by returning a malicious URL in its response. Different backbone LLMs were examined, and we found such cyber attacks can succeed in agents powered by most mainstream LLMs, with the reasoning models such as DeepSeek-R1 being the most vulnerable.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 24 of its 35 references that are in the map. First posted on arXiv 2 Apr 2025.

Builds on

24 of 35 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents11 Oct
  2. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents3 Oct
  3. 2024EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakage17 Sep
  4. 2024AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases17 Jul
  5. 2024Dissecting Adversarial Robustness of Multimodal LM Agents18 Jun
  6. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents5 Mar
  7. 2024Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fast13 Feb

Other references · 17

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  3. 2024GPT-4o System Cardcited by 4,588
  4. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  5. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 785
  6. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 486
  7. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Promptscited by 398
  8. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Modelscited by 328
  9. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Modelscited by 273
  10. 2024Agent Hospital: A Simulacrum of Hospital with Evolvable Medical Agentscited by 218
  11. 2023Almanac: Retrieval-Augmented Language Models for Clinical Medicinecited by 123
  12. 2023Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for Large Language Modelscited by 110
5 more
  1. 2024MMedAgent: Learning to Use Medical Tools with Multi-modal Agentcited by 109
  2. 2025MedRAX: Medical Reasoning Agent for Chest X-raycited by 70
  3. 2023Large AI Models in Health Informatics: Applications, Challenges, and the Futurecited by 61
  4. 2024One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Modelscited by 48
  5. 2024MEDCO: Medical Education Copilots Based on A Multi-Agent Frameworkcited by 33

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 1.

8 papers.

  1. 2025Safety in Large Reasoning Models: A Surveycited by 64
  2. 2025MedSentry: Understanding and Mitigating Safety Risks in Medical LLM Multi-Agent Systemscited by 9
  3. 2025SynapseRoute: An Auto-Route Switching Framework on Dual-State Large Language Modelcited by 7
  4. 2025When Models Outthink Their Safety: Unveiling and Mitigating Self-Jailbreak in Large Reasoning Modelscited by 6
  5. 2025SAS: Simulated Attention Scorecited by 4
  6. 2025Benchmarking the Thinking Mode of Multimodal Large Language Models in Clinical Taskscited by 2
  7. 2025Diagnosing Shoulder Disorders Using Multimodal Large Language Models and Consumer-Grade Camerascited by 1
  8. 2026ReasoningLens: Hierarchical Visualization and Diagnostic Auditing for Large Reasoning Modelscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Towards Safe AI Clinicians: A Comprehensive Study on Large Language Model Jailbreaking in Healthcare
  2. 2024When LLMs Go Online: The Emerging Threat of Web-Enabled LLMs
  3. 2025Why Are Web AI Agents More Vulnerable Than Standalone LLMs? A Security Analysis
  4. 2025Many-to-One Adversarial Consensus: Exposing Multi-Agent Collusion Risks in AI-Based Healthcare
  5. 2024Prompt Infection: LLM-to-LLM Prompt Injection within Multi-Agent Systems
  6. 2025A Practical Framework for Evaluating Medical AI Security: Reproducible Assessment of Jailbreaking and Privacy Vulnerabilities Across Clinical Specialties
  7. 2026Detection of adversarial intent in Human-AI teams using LLMs
  8. 2025The Hidden Dangers of Browsing AI Agents