mlsanityData to 7 Oct 2026

Effective Prompt Extraction from Language Models

arXiv 2307.06865 ↗First posted 13 Jul 2023cs.CL cs.AIcited by 83

Not on the map. Why?

Cited by 29 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The text generated by large language models is commonly controlled by prompting, where a prompt prepended to a user's query guides the model's output. The prompts used by companies to guide their models are often treated as secrets, to be hidden from the user making the query. They have even been treated as commodities to be bought and sold on marketplaces. However, anecdotal reports have shown adversarial users employing prompt extraction attacks to recover these prompts. In this paper, we present a framework for systematically measuring the effectiveness of these attacks. In experiments with 3 different sources of prompts and 11 underlying large language models, we find that simple text-based attacks can in fact reveal prompts with high probability. Our framework determines with high precision whether an extracted prompt is the actual secret prompt, rather than a model hallucination. Prompt extraction from real systems such as Claude 3 and ChatGPT further suggest that system prompts can be revealed by an adversary despite existing defenses in place.

Key facts. Cited by 83 papers in the map (data through 7 Oct 2026). Builds on 27 of its 46 references that are in the map. First posted on arXiv 13 Jul 2023.

Builds on

27 of 46 references.

References · 27

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  9. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  10. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  11. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  12. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
15 more
  1. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  2. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  3. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  4. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  5. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  6. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  7. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  8. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  9. 2023Prompt Injection attack against LLM-integrated Applicationscited by 568
  10. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  11. 2021How Many Data Points is a Prompt Worth?cited by 197
  12. 2022Preventing Verbatim Memorization in Language Models Gives a False Sense of Privacycited by 97
  13. 2020Trading Off Diversity and Quality in Natural Language Generationcited by 94
  14. 2023Black Box Adversarial Prompting for Foundation Modelscited by 62
  15. 2023INSTRUCTEVAL: Towards Holistic Evaluation of Instruction-Tuned Large Language Modelscited by 62

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 26; 2025: 32; 2026: 21.

83 papers.

Sort:

By area:

  1. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attackscited by 189
  2. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  3. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 122
  4. 2023AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Modelscited by 102
  5. 2024Machine Against the RAG: Jamming Retrieval-Augmented Generation with Blocker Documentscited by 93
  6. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agentscited by 63
  7. 2023Jailbreaking GPT-4V via Self-Adversarial Attacks with System Promptscited by 62
  8. 2024Instructional Segment Embedding: Improving LLM Safety with Instruction Hierarchycited by 62
  9. 2024Persistent Pre-Training Poisoning of LLMscited by 46
  10. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 44
  11. 2024Extracting Prompts by Inverting LLM Outputscited by 40
  12. 2025Embodied-R: Collaborative Framework for Activating Embodied Spatial Reasoning in Foundation Models via Reinforcement Learningcited by 37
38 more
  1. 2024Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirementscited by 35
  2. 2024Privacy Leakage on DNNs: A Survey of Model Inversion Attacks and Defensescited by 30
  3. 2024Prompt Leakage effect and defense strategies for multi-turn LLM interactionscited by 23
  4. 2024Fantastic Copyrighted Beasts and How (Not) to Generate Themcited by 23
  5. 2025ASIDE: Architectural Separation of Instructions and Data in Language Modelscited by 21
  6. 2025MCPXKIT: The Unified Toolkit for Analyzing Model Context Protocol Securitycited by 21
  7. 2024A Study on Prompt Injection Attack Against LLM-Integrated Mobile Robotic Systemscited by 15
  8. 2024Model Inversion Attacks: A Survey of Approaches and Countermeasurescited by 11
  9. 2024Prompt Obfuscation for Large Language Modelscited by 10
  10. 2024PRSA: Prompt Stealing Attacks against Real-World Prompt Servicescited by 9
  11. 2024Iteratively Prompting Multimodal LLMs to Reproduce Natural and AI-Generated Imagescited by 9
  12. 2025Better Language Model Inversion by Compactly Representing Next-Token Distributionscited by 9
  13. 2024Why Are My Prompts Leaked? Unraveling Prompt Extraction Threats in Customized Large Language Modelscited by 8
  14. 2025IP Leakage Attacks Targeting LLM-Based Multi-Agent Systemscited by 8
  15. 2024SOS! Soft Prompt Attack Against Open-Source Large Language Modelscited by 7
  16. 2025Federated In-Context Learning: Iterative Refinement for Improved Answer Qualitycited by 7
  17. 2025Diffusion LLMs are Natural Adversaries for any LLMcited by 7
  18. 2024A Jailbroken GenAI Model Can Cause Substantial Harm: GenAI-powered Applications are Vulnerable to PromptWarescited by 6
  19. 2025When AI Meets the Web: Prompt Injection Risks in Third-Party AI Chatbot Pluginscited by 6
  20. 2026Many-Tier Instruction Hierarchy in LLM Agentscited by 6
  21. 2025DP-Fusion: Token-Level Differentially Private Inference for Large Language Modelscited by 5
  22. 2025Network-Level Prompt and Trait Leakage in Local Research Agentscited by 5
  23. 2024Reconstruction of Differentially Private Text Sanitization via Large Language Modelscited by 4
  24. 2024Jailbreaking LLM-Controlled Robotscited by 4
  25. 2024Towards Action Hijacking of Large Language Model-based Agentcited by 4
  26. 2024PromptKeeper: Safeguarding System Prompts for LLMscited by 4
  27. 2025Chain-of-Trigger: An Agentic Backdoor that Paradoxically Enhances Agentic Robustnesscited by 4
  28. 2025ContextLeak: Auditing Leakage in Private In-Context Learning Methodscited by 4
  29. 2026Agent Skills Matter: Inferring Proprietary Skills from Execution Trajectoriescited by 4
  30. 2023Reverse-Engineering Decoding Strategies Given Blackbox Access to a Language Generation Systemcited by 3
  31. 2025Has My System Prompt Been Used? Large Language Model Prompt Membership Inferencecited by 3
  32. 2025Automating Prompt Leakage Attacks on Large Language Models Using Agentic Approachcited by 3
  33. 2025Evaluating Language Model Reasoning about Confidential Informationcited by 3
  34. 2025Every Language Model Has a Forgery-Resistant Signaturecited by 3
  35. 2026Just Ask: Curious Code Agents Reveal System Prompts in Frontier LLMscited by 3
  36. 2026TrinityGuard: A Unified Framework for Safeguarding Multi-Agent Systemscited by 3
  37. 2026Measuring Real-World Prompt Injection Attacks in LLM-based Resume Screeningcited by 3
  38. 2024Large Language Models as Carriers of Hidden Messagescited by 2

50 most cited of 83 All 83 (JSON)