mlsanityData to 14 Oct 2026

Lessons from Defending Gemini Against Indirect Prompt Injections

arXiv 2505.14534 ↗First posted 20 May 2025cs.CR cs.LGcited by 43

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

Gemini is increasingly used to perform tasks on behalf of users, where function-calling and tool-use capabilities enable the model to access user data. Some tools, however, require access to untrusted data introducing risk. Adversaries can embed malicious instructions in untrusted data which cause the model to deviate from the user's expectations and mishandle their data or permissions. In this report, we set out Google DeepMind's approach to evaluating the adversarial robustness of Gemini models and describe the main lessons learned from the process. We test how Gemini performs against a sophisticated adversary through an adversarial evaluation framework, which deploys a suite of adaptive attack techniques to run continuously against past, current, and future versions of Gemini. We describe how these ongoing evaluations directly help make Gemini more resilient against manipulation.

Key facts. Cited by 43 papers in the map (data through 14 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. First posted on arXiv 20 May 2025.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 20; 2026: 23.

43 papers.

Sort:

By area:

  1. 2025The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injectionscited by 129
  2. 2026AttriGuard: Defeating Indirect Prompt Injection in LLM Agents via Causal Attribution of Tool Invocationscited by 38
  3. 2025Defending Against Prompt Injection with DataFiltercited by 35
  4. 2025PromptLocate: Localizing Prompt Injection Attackscited by 28
  5. 2026Hidden in Memory: Sleeper Memory Poisoning in LLM Agentscited by 27
  6. 2026Poison Once, Exploit Forever: Environment-Injected Memory Poisoning Attacks on Web Agentscited by 24
  7. 2025ObliInjection: Order-Oblivious Prompt Injection Attack to LLM Agents with Multi-source Datacited by 18
  8. 2026CaMeLs Can Use Computers Too: System-level Security for Computer Use Agentscited by 17
  9. 2025Adversarial Reinforcement Learning for Large Language Model Agent Safetycited by 16
  10. 2025ceLLMate: Sandboxing Browser AI Agentscited by 16
  11. 2026Overcoming the Retrieval Barrier: Indirect Prompt Injection in the Wild for LLM Systemscited by 16
  12. 2026PISmith: Reinforcement Learning-based Red Teaming for Prompt Injection Defensescited by 16
31 more
  1. 2025May I have your Attention? Breaking Fine-Tuning based Prompt Injection Defenses using Architecture-Aware Attackscited by 15
  2. 2025A Safety and Security Framework for Real-World Agentic Systemscited by 13
  3. 2025Breaking and Fixing Defenses Against Control-Flow Hijacking in Multi-Agent Systemscited by 12
  4. 2025Meta SecAlign: A Secure Foundation LLM Against Prompt Injection Attackscited by 10
  5. 2025Web Fraud Attacks Against LLM-Driven Multi-Agent Systemscited by 10
  6. 2025SecInfer: Preventing Prompt Injection via Inference-time Scalingcited by 10
  7. 2025When Developer Aid Becomes Security Debt: A Systematic Analysis of Insecure Behaviors in LLM Coding Agentscited by 8
  8. 2026GPT-Red: Automated Red Teaming via Self-Play at Scalecited by 8
  9. 2025Bypassing Prompt Guards in Production with Controlled-Release Promptingcited by 7
  10. 2026Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluationcited by 7
  11. 2026Towards trustworthy agentic AI: a comprehensive survey of safety, robustness, privacy, and system securitycited by 5
  12. 2026Measuring Real-World Prompt Injection Attacks in LLM-based Resume Screeningcited by 5
  13. 2025Penetration Testing of Agentic AI: A Comparative Security Analysis Across Models and Frameworkscited by 4
  14. 2026Gram: Assessing sabotage propensities via automated alignment auditingcited by 4
  15. 2025Multi-Stage Prompt Inference Attacks on Enterprise LLM Systemscited by 3
  16. 2025Exploiting Web Search Tools of AI Agents for Data Exfiltrationcited by 3
  17. 2025PLAGUE: Plug-and-play framework for Lifelong Adaptive Generation of Multi-turn Exploitscited by 3
  18. 2025Soft Instruction De-escalation Defensecited by 3
  19. 2026Hijacking Large Audio-Language Models via Context-Agnostic and Imperceptible Auditory Prompt Injectioncited by 3
  20. 2025Rule Encoding and Compliance in Large Language Models: An Information-Theoretic Analysiscited by 2
  21. 2026Dual-Modality Multi-Stage Adversarial Safety Training: Robustifying Multimodal Web Agents Against Cross-Modal Attackscited by 2
  22. 2026Protecting Context and Prompts: Deterministic Security for Non-Deterministic AIcited by 1
  23. 2026Stand-Alone Complex or Vibercrime? Exploring the adoption and innovation of GenAI tools, coding assistants, and agents within cybercrime ecosystemscited by 0
  24. 2026FlashRT: Towards Computationally and Memory Efficient Red-Teaming for Prompt Injection and Knowledge Corruptioncited by 0
  25. 2026How Useful Is Cross-Domain Generalization for Training LLM Monitors?cited by 0
  26. 2026PromptShield Home: Ambient Multimodal Prompt Injection Defense for Smart-Home Agentscited by 0
  27. 2026On Understanding, Identifying, and Mitigating Vulnerabilities in Agentic Large Language Modelscited by 0
  28. 2026When the Agent Becomes the Kernel: A Systematization of Security on the Path to AI-Native Operating Systemscited by 0
  29. 2026ToolFence: Fine-Grained Authorization for Secure Tool-Using LLM Agentscited by 0
  30. 2026AdvSim2Real : Training Web Agents Against Adaptive Prompt Injection in a Web World Modelcited by 0
  31. 2026Prompt-Injection Probes Learn the Conversation, Not Just the Injectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Fun-tuning: Characterizing the Vulnerability of Proprietary LLMs to Optimization-based Prompt Injection Attacks via the Fine-Tuning Interface
  2. 2023Tensor Trust: Interpretable Prompt Injection Attacks from an Online Game
  3. 2026Defense Against Indirect Prompt Injection via Tool Result Parsing
  4. 2026The Granularity Gap: A Multi-Dimensional Cross-Generational Audit of Sycophancy in Gemini Models
  5. 2026Building Production-Ready Probes For Gemini
  6. 2026Evaluation of Prompt Injection Defenses in Large Language Models
  7. 2025Adversarial Prompt Evaluation: Systematic Benchmarking of Guardrails Against Prompt Input Attacks on LLMs
  8. 2025Evaluating Adversarial Vulnerabilities in Modern Large Language Models