mlsanityData to 14 Oct 2026

PLeak: Prompt Leaking Attacks against Large Language Model Applications

arXiv 2405.06823 ↗First posted 10 May 2024cs.CR cs.AI cs.LGcited by 127

Not on the map. Why?

Cited by 67 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large Language Models (LLMs) enable a new ecosystem with many downstream applications, called LLM applications, with different natural language processing tasks. The functionality and performance of an LLM application highly depend on its system prompt, which instructs the backend LLM on what task to perform. Therefore, an LLM application developer often keeps a system prompt confidential to protect its intellectual property. As a result, a natural attack, called prompt leaking, is to steal the system prompt from an LLM application, which compromises the developer's intellectual property. Existing prompt leaking attacks primarily rely on manually crafted queries, and thus achieve limited effectiveness. In this paper, we design a novel, closed-box prompt leaking attack framework, called PLeak, to optimize an adversarial query such that when the attacker sends it to a target LLM application, its response reveals its own system prompt. We formulate finding such an adversarial query as an optimization problem and solve it with a gradient-based method approximately. Our key idea is to break down the optimization goal by optimizing adversary queries for system prompts incrementally, i.e., starting from the first few tokens of each system prompt step by step until the entire length of the system prompt. We evaluate PLeak in both offline settings and for real-world LLM applications, e.g., those on Poe, a popular platform hosting such applications. Our results show that PLeak can effectively leak system prompts and significantly outperforms not only baselines that manually curate queries but also baselines with optimized queries that are modified and adapted from existing jailbreaking attacks. We responsibly reported the issues to Poe and are still waiting for their response. Our implementation is available at this repository: https://github.com/BHui97/PLeak.

Key facts. Cited by 127 papers in the map (data through 14 Oct 2026). Builds on 31 of its 61 references that are in the map. First posted on arXiv 10 May 2024.

Builds on

31 of 61 references.

References · 31

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2017Deep reinforcement learning from human preferencescited by 3,912
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  7. 2020Extracting Training Data from Large Language Modelscited by 2,189
  8. 2019Language Models as Knowledge Bases?cited by 1,936
  9. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  10. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
  11. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  12. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
19 more
  1. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  2. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  3. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 832
  4. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 785
  5. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  6. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 722
  7. 2021GPT Understands, Toocited by 716
  8. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 651
  9. 2016Stealing Machine Learning Models via Prediction APIscited by 619
  10. 20218-bit Optimizers via Block-wise Quantizationcited by 403
  11. 2023Formalizing and Benchmarking Prompt Injection Attacks and Defensescited by 364
  12. 2017Beam Search Strategies for Neural Machine Translationcited by 232
  13. 2018Stealing Hyperparameters in Machine Learningcited by 118
  14. 2020Stealing Links from Graph Neural Networkscited by 115
  15. 2020How Can We Accelerate Progress Towards Human-like Linguistic Generalization?cited by 114
  16. 2023You Only Prompt Once: On the Capabilities of Prompt Learning on Large Language Models to Tackle Toxic Contentcited by 39
  17. 2023In ChatGPT We Trust? Measuring and Characterizing the Reliability of ChatGPTcited by 37
  18. 2023Stealing the Decoding Algorithms of Language Modelscited by 19
  19. 2021What Do You Get When You Cross Beam Search with Nucleus Sampling?cited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 19; 2025: 62; 2026: 46.

127 papers.

Sort:

By area:

  1. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 205
  2. 2025Prompt Injection Attack to Tool Selection in LLM Agentscited by 158
  3. 2025DataSentinel: A Game-Theoretic Detection of Prompt Injection Attackscited by 143
  4. 2025Unveiling Privacy Risks in LLM Agent Memorycited by 101
  5. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agentscited by 65
  6. 2025WebInject: Prompt Injection Attack to Web Agentscited by 51
  7. 2024LLM-PBE: Assessing Data Privacy in Large Language Modelscited by 41
  8. 2024Feedback-Guided Extraction of Knowledge Base from Retrieval-Augmented LLM Applicationscited by 37
  9. 2025A Critical Evaluation of Defenses against Prompt Injection Attackscited by 36
  10. 2024Unleashing Worms and Extracting Data: Escalating the Outcome of Attacks against RAG-based Inference in Scale and Severity Using Jailbreakingcited by 30
  11. 2025Security Concerns for Large Language Models: A Surveycited by 29
  12. 2026AgentSentry: Mitigating Indirect Prompt Injection in LLM Agents via Temporal Causal Diagnostics and Context Purificationcited by 29
38 more
  1. 2025PromptLocate: Localizing Prompt Injection Attackscited by 28
  2. 2025AgentSentinel: An End-to-End and Real-Time Security Defense Framework for Computer-Use Agentscited by 27
  3. 2024Large Model Based Agents: State-of-the-Art, Cooperation Paradigms, Security and Privacy, and Future Trendscited by 26
  4. 2025Riddle Me This! Stealthy Membership Inference for Retrieval-Augmented Generationcited by 26
  5. 2025The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMscited by 26
  6. 2025ASIDE: Architectural Separation of Instructions and Data in Language Modelscited by 24
  7. 2024Enhancing Prompt Injection Attacks to LLMs via Poisoning Alignmentcited by 21
  8. 2025PIShield: Detecting Prompt Injection Attacks via Intrinsic LLM Featurescited by 20
  9. 2025A Survey on Model Extraction Attacks and Defenses for Large Language Modelscited by 19
  10. 2025ObliInjection: Order-Oblivious Prompt Injection Attack to LLM Agents with Multi-source Datacited by 18
  11. 2024PROMPTFUZZ: Harnessing Fuzzing Techniques for Robust Testing of Prompt Injection in LLMscited by 17
  12. 2026VIGIL: Defending LLM Agents Against Tool Stream Injection via Verify-Before-Commitcited by 17
  13. 2026A Framework for Formalizing LLM Agent Securitycited by 16
  14. 2025"Your AI, My Shell": Demystifying Prompt Injection Attacks on Agentic AI Coding Editorscited by 15
  15. 2025The Pitfalls of KV Cache Compressioncited by 15
  16. 2024On the (In)Security of LLM App Storescited by 14
  17. 2024LeakAgent: RL-based Red-teaming Agent for LLM Privacy Leakagecited by 13
  18. 2025ControlNET: A Firewall for RAG-based LLM Systemcited by 12
  19. 2025How Not to Detect Prompt Injections with an LLMcited by 12
  20. 2025Towards Lifecycle Unlearning Commitment Management: Measuring Sample-level Unlearning Completenesscited by 11
  21. 2026PIArena: A Platform for Prompt Injection Evaluationcited by 11
  22. 2025ADMIT: Few-shot Knowledge Poisoning Attacks on RAG-based Fact Checkingcited by 10
  23. 2025Attention is All You Need to Defend Against Indirect Prompt Injection Attacks in LLMscited by 10
  24. 2024Recent advancements in LLM Red-Teaming: Techniques, Defenses, and Ethical Considerationscited by 8
  25. 2025LLM Applications: Current Paradigms and the Next Frontiercited by 8
  26. 2025IP Leakage Attacks Targeting LLM-Based Multi-Agent Systemscited by 8
  27. 2025A Taxonomy of Prompt Defects in LLM Systemscited by 8
  28. 2025Fingerprinting LLMs via Prompt Injectioncited by 8
  29. 2024GenTel-Safe: A Unified Benchmark and Shielding Framework for Defending Against Prompt Injection Attackscited by 7
  30. 2025ARGUS: Defending Against Multimodal Indirect Prompt Injection via Steering Instruction-Following Behaviorcited by 7
  31. 2025Beyond Data Privacy: New Privacy Risks for Large Language Modelscited by 6
  32. 2026Towards Security-Auditable LLM Agents: A Unified Graph Representationcited by 6
  33. 2024Systematic Categorization, Construction and Evaluation of New Attacks against Multi-modal Mobile GUI Agentscited by 5
  34. 2024Enhance Modality Robustness in Text-Centric Multimodal Alignment with Adversarial Promptingcited by 5
  35. 2024PromptKeeper: Safeguarding System Prompts for LLMscited by 5
  36. 2025Bleeding Pathways: Vanishing Discriminability in LLM Hidden States Fuels Jailbreak Attackscited by 5
  37. 2025Network-Level Prompt and Trait Leakage in Local Research Agentscited by 5
  38. 2025Red-Teaming Coding Agents from a Tool-Invocation Perspective: An Empirical Security Assessmentcited by 5

50 most cited of 127 All 127 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Breaking to Build: A Threat Model of Prompt-Based Attacks for Securing LLMs
  2. 2025Automating Prompt Leakage Attacks on Large Language Models Using Agentic Approach
  3. 2024SoK: Prompt Hacking of Large Language Models
  4. 2024Exploring Vulnerabilities and Protections in Large Language Models: A Survey
  5. 2025Fun-tuning: Characterizing the Vulnerability of Proprietary LLMs to Optimization-based Prompt Injection Attacks via the Fine-Tuning Interface
  6. 2024Automatic and Universal Prompt Injection Attacks against Large Language Models
  7. 2025Paper Summary Attack: Jailbreaking LLMs through LLM Safety Papers
  8. 2026An Empirical Evaluation of Prompt Injection Vulnerabilities in Large Language Models Across Multilingual and Obfuscated Attack Scenarios