mlsanityData to 14 Oct 2026

An Empirical Study of Privacy Leakage Chains via Prompt Injection in Black-Box Chatbot Environments

arXiv 2605.18133 ↗First posted 18 May 2026cs.CR cs.AI cs.HCcited by 0

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

LLM-based chatbot agents increasingly process user requests by combining natural-language reasoning with external tools such as web browsing. These capabilities improve usability, but they also create attack surfaces when untrusted external content is processed as part of a user' s task. This paper studies a privacy-leakage attack chain based on indirect prompt injection in black-box chatbot environments, where the attacker has no access to model weights, system prompts, or agent implementation details including how a trajectory is actually managed during its processing for a query. We first analyze how an attacker can hijack an agent' s intended task by crafting external content that appears benign to the victim while inducing the agent to execute an attacker-defined objective. We then evaluate a new prompt-injection technique, called exemplification, which uses a bridge in the external content to reframe the user prompt and the benign beginning of the retrieved page as few-shot examples before appending the attacker' s objective. We compare its attack success rate with a prior fake-completion technique. Finally, we demonstrate a proof-of-concept data-exfiltration chain using fictitious personal information in a controlled setting. Our results suggest that prompt injection, jailbreak-style instruction steering, and web-tool invocation can be combined into a feasible privacy-leakage path in deployed chatbot agents.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 36 of its 41 references that are in the map. First posted on arXiv 18 May 2026.

Builds on

36 of 41 references.

Closest ancestors · 14

Same direction, newest first

  1. 2026ARGUS: Defending LLM Agents Against Context-Aware Prompt Injection5 May
  2. 2026AgentSentry: Mitigating Indirect Prompt Injection in LLM Agents via Temporal Causal Diagnostics and Context Purification26 Feb
  3. 2026Silent Egress: When Implicit Prompt Injection Makes LLM Agents Leak Without a Trace25 Feb
  4. 2025EchoLeak: The First Real-World Zero-Click Prompt Injection Exploit in a Production LLM System6 Sep
  5. 2025PromptArmor: Simple yet Effective Prompt Injection Defenses21 Jul
  6. 2025Simple Prompt Injection Attacks Can Leak Personal Data Observed by LLM Agents During Task Execution1 Jun
  7. 2024System-Level Defense against Indirect Prompt Injection Attacks: An Information Flow Control Perspective27 Sep
  8. 2024AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases17 Jul
  9. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions19 Apr
  10. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents5 Mar
  11. 2024StruQ: Defending Against Prompt Injection with Structured Queries9 Feb
  12. 2024Signed-Prompt: A New Approach to Prevent Prompt Injection Attacks Against LLM-Integrated Applications15 Jan
2 more
  1. 2023Jatmo: Prompt Injection Defense by Task-Specific Finetuning29 Dec
  2. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection23 Feb

Other references · 22

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  4. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  5. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  6. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  7. 2020Extracting Training Data from Large Language Modelscited by 2,189
  8. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  9. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  10. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  11. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  12. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 832
10 more
  1. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 785
  2. 2022Quantifying Memorization Across Neural Language Modelscited by 763
  3. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 591
  4. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 560
  5. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 434
  6. 2023Tool Learning with Foundation Modelscited by 302
  7. 2024PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Modelscited by 267
  8. 2023TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIscited by 184
  9. 2022MRKL Systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoningcited by 156
  10. 2024Exfiltration of personal information from ChatGPT via prompt injectioncited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026PromptPET: Privacy-Utility Optimized Prompt Obfuscation
  2. 2026Confuse the Model, Control the Flow: Understanding and Mitigating Privacy Leakage from LLM Agents with Information Flow Control
  3. 2025When AI Meets the Web: Prompt Injection Risks in Third-Party AI Chatbot Plugins
  4. 2026A Layered Security Framework Against Prompt Injection in RAG-Based Chatbots
  5. 2025ChatInject: Abusing Chat Templates for Prompt Injection in LLM Agents
  6. 2026The Innocent Courier: Covert Exfiltration Through Legitimate LLM Web Fetching
  7. 2024Imprompter: Tricking LLM Agents into Improper Tool Use
  8. 2024Whispers in the Machine: Confidentiality in Agentic Systems