# Prompt Injection Attacks on Large Language Models in Oncology

arXiv: https://arxiv.org/abs/2407.18981 · HTML: https://mlsanity.com/paper/2407.18981/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security.
- First posted on arXiv 23 Jul 2024.

## Placed in

- LLM safety and security / [Safety alignment, refusal and guardrails](https://mlsanity.com/safety/safety-alignment-refusal-and-guardrails/)

## Built on by (most cited first, 8 of 8)

- [The Landscape of Prompt Injection Threats in LLM Agents: From Taxonomy to Analysis](https://mlsanity.com/paper/2602.10453/) (2026)
- [MedSentry: Understanding and Mitigating Safety Risks in Medical LLM Multi-Agent Systems](https://mlsanity.com/paper/2505.20824/) (2025)
- [Palisade -- Prompt Injection Detection Framework](https://mlsanity.com/paper/2410.21146/) (2024)
- [Beyond Multiple-Choice Accuracy: Real-World Challenges of Implementing Large Language Models in Healthcare](https://mlsanity.com/paper/2410.18460/) (2024)
- [Semantic-Preserving Prompt Hijacking: A Black-Box Adversarial Attack on Auto-Prompt Optimization](https://mlsanity.com/paper/2506.18756/) (2025)
- [Manipulating Transformer-Based Models: Controllability, Steerability, and Robust Interventions](https://mlsanity.com/paper/2509.04549/) (2025)
- [Goal-Driven Risk Assessment for LLM-Powered Systems: A Healthcare Case Study](https://mlsanity.com/paper/2603.03633/) (2026)
- [Where Do LLM-based Systems Break? A System-Level Security Framework for Risk Assessment and Treatment](https://mlsanity.com/paper/2603.07460/) (2026)

## Abstract (arXiv)

Vision-language artificial intelligence models (VLMs) possess medical knowledge and can be employed in healthcare in numerous ways, including as image interpreters, virtual scribes, and general decision support systems. However, here, we demonstrate that current VLMs applied to medical tasks exhibit a fundamental security flaw: they can be attacked by prompt injection attacks, which can be used to output harmful information just by interacting with the VLM, without any access to its parameters. We performed a quantitative study to evaluate the vulnerabilities to these attacks in four state of the art VLMs which have been proposed to be of utility in healthcare: Claude 3 Opus, Claude 3.5 Sonnet, Reka Core, and GPT-4o. Using a set of N=297 attacks, we show that all of these models are susceptible. Specifically, we show that embedding sub-visual prompts in medical imaging data can cause the model to provide harmful output, and that these prompts are non-obvious to human observers. Thus, our study demonstrates a key vulnerability in medical VLMs which should be mitigated before widespread clinical adoption.