# Lessons from Defending Gemini Against Indirect Prompt Injections

arXiv: https://arxiv.org/abs/2505.14534 · HTML: https://mlsanity.com/paper/2505.14534/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security.
- First posted on arXiv 20 May 2025.

## Placed in

- LLM safety and security / [Prompt injection and agent security](https://mlsanity.com/safety/prompt-injection-and-agent-security/)

## Built on by (most cited first, 30 of 43)

- [The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injections](https://mlsanity.com/paper/2510.09023/) (2025)
- [AttriGuard: Defeating Indirect Prompt Injection in LLM Agents via Causal Attribution of Tool Invocations](https://mlsanity.com/paper/2603.10749/) (2026)
- [Defending Against Prompt Injection with DataFilter](https://mlsanity.com/paper/2510.19207/) (2025)
- [PromptLocate: Localizing Prompt Injection Attacks](https://mlsanity.com/paper/2510.12252/) (2025)
- [Hidden in Memory: Sleeper Memory Poisoning in LLM Agents](https://mlsanity.com/paper/2605.15338/) (2026)
- [Poison Once, Exploit Forever: Environment-Injected Memory Poisoning Attacks on Web Agents](https://mlsanity.com/paper/2604.02623/) (2026)
- [ObliInjection: Order-Oblivious Prompt Injection Attack to LLM Agents with Multi-source Data](https://mlsanity.com/paper/2512.09321/) (2025)
- [CaMeLs Can Use Computers Too: System-level Security for Computer Use Agents](https://mlsanity.com/paper/2601.09923/) (2026)
- [Adversarial Reinforcement Learning for Large Language Model Agent Safety](https://mlsanity.com/paper/2510.05442/) (2025)
- [ceLLMate: Sandboxing Browser AI Agents](https://mlsanity.com/paper/2512.12594/) (2025)
- [Overcoming the Retrieval Barrier: Indirect Prompt Injection in the Wild for LLM Systems](https://mlsanity.com/paper/2601.07072/) (2026)
- [PISmith: Reinforcement Learning-based Red Teaming for Prompt Injection Defenses](https://mlsanity.com/paper/2603.13026/) (2026)
- [May I have your Attention? Breaking Fine-Tuning based Prompt Injection Defenses using Architecture-Aware Attacks](https://mlsanity.com/paper/2507.07417/) (2025)
- [A Safety and Security Framework for Real-World Agentic Systems](https://mlsanity.com/paper/2511.21990/) (2025)
- [Breaking and Fixing Defenses Against Control-Flow Hijacking in Multi-Agent Systems](https://mlsanity.com/paper/2510.17276/) (2025)
- [Meta SecAlign: A Secure Foundation LLM Against Prompt Injection Attacks](https://mlsanity.com/paper/2507.02735/) (2025)
- [Web Fraud Attacks Against LLM-Driven Multi-Agent Systems](https://mlsanity.com/paper/2509.01211/) (2025)
- [SecInfer: Preventing Prompt Injection via Inference-time Scaling](https://mlsanity.com/paper/2509.24967/) (2025)
- [When Developer Aid Becomes Security Debt: A Systematic Analysis of Insecure Behaviors in LLM Coding Agents](https://mlsanity.com/paper/2507.09329/) (2025)
- [GPT-Red: Automated Red Teaming via Self-Play at Scale](https://mlsanity.com/paper/2607.26115/) (2026)
- [Bypassing Prompt Guards in Production with Controlled-Release Prompting](https://mlsanity.com/paper/2510.01529/) (2025)
- [Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluation](https://mlsanity.com/paper/2606.10749/) (2026)
- [Towards trustworthy agentic AI: a comprehensive survey of safety, robustness, privacy, and system security](https://mlsanity.com/paper/2605.23989/) (2026)
- [Measuring Real-World Prompt Injection Attacks in LLM-based Resume Screening](https://mlsanity.com/paper/2605.28999/) (2026)
- [Penetration Testing of Agentic AI: A Comparative Security Analysis Across Models and Frameworks](https://mlsanity.com/paper/2512.14860/) (2025)
- [Gram: Assessing sabotage propensities via automated alignment auditing](https://mlsanity.com/paper/2605.30322/) (2026)
- [Multi-Stage Prompt Inference Attacks on Enterprise LLM Systems](https://mlsanity.com/paper/2507.15613/) (2025)
- [Exploiting Web Search Tools of AI Agents for Data Exfiltration](https://mlsanity.com/paper/2510.09093/) (2025)
- [PLAGUE: Plug-and-play framework for Lifelong Adaptive Generation of Multi-turn Exploits](https://mlsanity.com/paper/2510.17947/) (2025)
- [Soft Instruction De-escalation Defense](https://mlsanity.com/paper/2510.21057/) (2025)

## Abstract (arXiv)

Gemini is increasingly used to perform tasks on behalf of users, where function-calling and tool-use capabilities enable the model to access user data. Some tools, however, require access to untrusted data introducing risk. Adversaries can embed malicious instructions in untrusted data which cause the model to deviate from the user's expectations and mishandle their data or permissions. In this report, we set out Google DeepMind's approach to evaluating the adversarial robustness of Gemini models and describe the main lessons learned from the process. We test how Gemini performs against a sophisticated adversary through an adversarial evaluation framework, which deploys a suite of adaptive attack techniques to run continuously against past, current, and future versions of Gemini. We describe how these ongoing evaluations directly help make Gemini more resilient against manipulation.