# Emerging Cyber Attack Risks of Medical AI Agents

arXiv: https://arxiv.org/abs/2504.03759 · HTML: https://mlsanity.com/paper/2504.03759/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 24 of its 35 references that are in the map.
- First posted on arXiv 2 Apr 2025.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 24 of 24)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models](https://mlsanity.com/paper/2310.04451/) (2023)
- [InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents](https://mlsanity.com/paper/2403.02691/) (2024)
- [Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks](https://mlsanity.com/paper/2404.02151/) (2024)
- [AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents](https://mlsanity.com/paper/2410.09024/) (2024)
- [AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases](https://mlsanity.com/paper/2407.12784/) (2024)
- [FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts](https://mlsanity.com/paper/2311.05608/) (2023)
- [Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents](https://mlsanity.com/paper/2410.02644/) (2024)
- [Visual Adversarial Examples Jailbreak Aligned Large Language Models](https://mlsanity.com/paper/2306.13213/) (2023)
- [Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models](https://mlsanity.com/paper/2307.14539/) (2023)
- [Agent Hospital: A Simulacrum of Hospital with Evolvable Medical Agents](https://mlsanity.com/paper/2405.02957/) (2024)
- [Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fast](https://mlsanity.com/paper/2402.08567/) (2024)
- [EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakage](https://mlsanity.com/paper/2409.11295/) (2024)
- [Almanac: Retrieval-Augmented Language Models for Clinical Medicine](https://mlsanity.com/paper/2303.01229/) (2023)
- [Dissecting Adversarial Robustness of Multimodal LM Agents](https://mlsanity.com/paper/2406.12814/) (2024)
- [Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for Large Language Models](https://mlsanity.com/paper/2305.14710/) (2023)
- [MMedAgent: Learning to Use Medical Tools with Multi-modal Agent](https://mlsanity.com/paper/2407.02483/) (2024)
- [MedRAX: Medical Reasoning Agent for Chest X-ray](https://mlsanity.com/paper/2502.02673/) (2025)
- [Large AI Models in Health Informatics: Applications, Challenges, and the Future](https://mlsanity.com/paper/2303.11568/) (2023)
- [One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Models](https://mlsanity.com/paper/2403.01849/) (2024)
- [MEDCO: Medical Education Copilots Based on A Multi-Agent Framework](https://mlsanity.com/paper/2408.12496/) (2024)

## Built on by (most cited first, 8 of 8)

- [Safety in Large Reasoning Models: A Survey](https://mlsanity.com/paper/2504.17704/) (2025)
- [MedSentry: Understanding and Mitigating Safety Risks in Medical LLM Multi-Agent Systems](https://mlsanity.com/paper/2505.20824/) (2025)
- [SynapseRoute: An Auto-Route Switching Framework on Dual-State Large Language Model](https://mlsanity.com/paper/2507.02822/) (2025)
- [When Models Outthink Their Safety: Unveiling and Mitigating Self-Jailbreak in Large Reasoning Models](https://mlsanity.com/paper/2510.21285/) (2025)
- [SAS: Simulated Attention Score](https://mlsanity.com/paper/2507.07694/) (2025)
- [Benchmarking the Thinking Mode of Multimodal Large Language Models in Clinical Tasks](https://mlsanity.com/paper/2511.03328/) (2025)
- [Diagnosing Shoulder Disorders Using Multimodal Large Language Models and Consumer-Grade Cameras](https://mlsanity.com/paper/2510.09230/) (2025)
- [ReasoningLens: Hierarchical Visualization and Diagnostic Auditing for Large Reasoning Models](https://mlsanity.com/paper/2606.23404/) (2026)

## Abstract (arXiv)

Large language models (LLMs)-powered AI agents exhibit a high level of autonomy in addressing medical and healthcare challenges. With the ability to access various tools, they can operate within an open-ended action space. However, with the increase in autonomy and ability, unforeseen risks also arise. In this work, we investigated one particular risk, i.e., cyber attack vulnerability of medical AI agents, as agents have access to the Internet through web browsing tools. We revealed that through adversarial prompts embedded on webpages, cyberattackers can: i) inject false information into the agent's response; ii) they can force the agent to manipulate recommendation (e.g., healthcare products and services); iii) the attacker can also steal historical conversations between the user and agent, resulting in the leak of sensitive/private medical information; iv) furthermore, the targeted agent can also cause a computer system hijack by returning a malicious URL in its response. Different backbone LLMs were examined, and we found such cyber attacks can succeed in agents powered by most mainstream LLMs, with the reasoning models such as DeepSeek-R1 being the most vulnerable.