mlsanityData to 7 Oct 2026

Assessing Prompt Injection Risks in 200+ Custom GPTs

arXiv 2311.11538 ↗First posted 20 Nov 2023cs.CR cs.AIcited by 49

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

In the rapidly evolving landscape of artificial intelligence, ChatGPT has been widely used in various applications. The new feature - customization of ChatGPT models by users to cater to specific needs has opened new frontiers in AI utility. However, this study reveals a significant security vulnerability inherent in these user-customized GPTs: prompt injection attacks. Through comprehensive testing of over 200 user-designed GPT models via adversarial prompts, we demonstrate that these systems are susceptible to prompt injections. Through prompt injection, an adversary can not only extract the customized system prompts but also access the uploaded files. This paper provides a first-hand analysis of the prompt injection, alongside the evaluation of the possible mitigation of such attacks. Our findings underscore the urgent need for robust security frameworks in the design and deployment of customizable GPT models. The intent of this paper is to raise awareness and prompt action in the AI community, ensuring that the benefits of GPT customization do not come at the cost of compromised security and privacy.

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 3 of its 6 references that are in the map. First posted on arXiv 20 Nov 2023.

Builds on

3 of 6 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Prompt Injection attack against LLM-integrated Applications8 Jun
  2. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection23 Feb

Other references · 1

  1. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 27; 2025: 17; 2026: 5.

49 papers.

Sort:

By area:

  1. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agentscited by 476
  2. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agentscited by 341
  3. 2024SecAlign: Defending Against Prompt Injection with Preference Optimizationcited by 179
  4. 2024Automatic and Universal Prompt Injection Attacks against Large Language Modelscited by 112
  5. 2024Follow My Instruction and Spill the Beans: Scalable Data Extraction from Retrieval-Augmented Generation Systemscited by 76
  6. 2024A New Era in LLM Security: Exploring Security Concerns in Real-World LLM-based Systemscited by 74
  7. 2025MELON: Provable Defense Against Indirect Prompt Injection Attacks in AI Agentscited by 67
  8. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agentscited by 63
  9. 2024Instructional Segment Embedding: Improving LLM Safety with Instruction Hierarchycited by 62
  10. 2024Signed-Prompt: A New Approach to Prevent Prompt Injection Attacks Against LLM-Integrated Applicationscited by 42
  11. 2024InjecGuard: Benchmarking and Mitigating Over-defense in Prompt Injection Guardrail Modelscited by 36
  12. 2024Feedback-Guided Extraction of Knowledge Base from Retrieval-Augmented LLM Applicationscited by 36
37 more
  1. 2026AttriGuard: Defeating Indirect Prompt Injection in LLM Agents via Causal Attribution of Tool Invocationscited by 33
  2. 2025A Closer Look at System Prompt Robustnesscited by 31
  3. 2024Mind the Inconspicuous: Revealing the Hidden Weakness in Aligned LLMs' Refusal Boundariescited by 26
  4. 2024Embedding-based classifiers can detect prompt injection attackscited by 24
  5. 2024GoEX: Perspectives and Designs Towards a Runtime for Autonomous LLM Applicationscited by 22
  6. 2024FATH: Authentication-based Test-time Defense against Indirect Prompt Injection Attackscited by 19
  7. 2024PROMPTFUZZ: Harnessing Fuzzing Techniques for Robust Testing of Prompt Injection in LLMscited by 17
  8. 2024Raccoon: Prompt Extraction Benchmark of LLM-Integrated Applicationscited by 16
  9. 2024A First Look at GPT Apps: Landscape and Vulnerabilitycited by 15
  10. 2025Doxing via the Lens: Revealing Location-related Privacy Leakage on Multi-modal Large Reasoning Modelscited by 14
  11. 2024LeakAgent: RL-based Red-teaming Agent for LLM Privacy Leakagecited by 13
  12. 2025Attack and defense techniques in large language models: A survey and new perspectivescited by 13
  13. 2025ControlNET: A Firewall for RAG-based LLM Systemcited by 11
  14. 2024Decoupled Alignment for Robust Plug-and-Play Adaptationcited by 9
  15. 2024Why Are My Prompts Leaked? Unraveling Prompt Extraction Threats in Customized Large Language Modelscited by 8
  16. 2025The Illusion of Role Separation: Hidden Shortcuts in LLM Role Learning (and How to Fix Them)cited by 6
  17. 2025POISONCRAFT: Practical Poisoning of Retrieval-Augmented Generation for Large Language Modelscited by 6
  18. 2025When AI Meets the Web: Prompt Injection Risks in Third-Party AI Chatbot Pluginscited by 6
  19. 2026Automating Agent Hijacking via Structural Template Injectioncited by 6
  20. 2024GPT Store Mining and Analysiscited by 5
  21. 2025You Can't Steal Nothing: Mitigating Prompt Leakages in LLMs via System Vectorscited by 4
  22. 2025RAG-Pull: Turning Retrieval into a Code-Injection Channel via Invisible Unicode Perturbationscited by 4
  23. 2025Encrypted Prompt: Securing LLM Applications Against Unauthorized Actionscited by 3
  24. 2024Too Big to Fool: Resisting Deception in Language Modelscited by 2
  25. 2025CachePrune: Teaching LLMs What Not to Follow via KV-Cache Editingcited by 2
  26. 2026Contrastive Reasoning Alignment: Reinforcement Learning from Hidden Representationscited by 2
  27. 2024Generative Models and Connected and Automated Vehicles: A Survey in Exploring the Intersection of Transportation and AIcited by 1
  28. 2024Can Large Language Models put 2 and 2 together? Probing for Entailed Arithmetical Relationshipscited by 1
  29. 2025An Empirically-grounded tool for Automatic Prompt Linting and Repair: A Case Study on Bias, Vulnerability, and Optimization in Developer Promptscited by 1
  30. 2025Towards Safer Chatbots: Automated Policy Compliance Evaluation of Custom GPTscited by 1
  31. 2026Talk is (Not) Cheap: A Taxonomy and Benchmark Coverage Audit for LLM Attackscited by 1
  32. 2024Privacy and Security Implications of Cloud-Based AI Services : A Surveycited by 0
  33. 2024Exploring the Privacy Protection Capabilities of Chinese Large Language Modelscited by 0
  34. 2025DMRL: Data- and Model-aware Reward Learning for Data Extractioncited by 0
  35. 2025When GPT Spills the Tea: Comprehensive Assessment of Knowledge File Leakage in GPTscited by 0
  36. 2025Provably Secure Retrieval-Augmented Generationcited by 0
  37. 2026Walking the Embedding Space: Datastore Extraction from Multimodal RAGcited by 0