mlsanityData to 7 Oct 2026

Ignore This Title and HackAPrompt: Exposing Systemic Vulnerabilities of LLMs through a Global Scale Prompt Hacking Competition

arXiv 2311.16119 ↗First posted 24 Oct 2023cs.CR cs.AI cs.CLcited by 65

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

Large Language Models (LLMs) are deployed in interactive contexts with direct user engagement, such as chatbots and writing assistants. These deployments are vulnerable to prompt injection and jailbreaking (collectively, prompt hacking), in which models are manipulated to ignore their original instructions and follow potentially malicious ones. Although widely acknowledged as a significant security threat, there is a dearth of large-scale resources and quantitative studies on prompt hacking. To address this lacuna, we launch a global prompt hacking competition, which allows for free-form human input attacks. We elicit 600K+ adversarial prompts against three state-of-the-art LLMs. We describe the dataset, which empirically verifies that current LLMs can indeed be manipulated via prompt hacking. We also present a comprehensive taxonomical ontology of the types of adversarial prompts.

Key facts. Cited by 65 papers in the map (data through 7 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 39 of its 71 references that are in the map. First posted on arXiv 24 Oct 2023.

Builds on

39 of 71 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Prompt Injection attack against LLM-integrated Applications8 Jun
  2. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection23 Feb
  3. 2022Red Teaming Language Models with Language Models7 Feb

Other references · 36

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  6. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  7. 2020Extracting Training Data from Large Language Modelscited by 2,144
  8. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  9. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  10. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  11. 2018Datasheets for Datasetscited by 1,371
  12. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
24 more
  1. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  2. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  3. 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 737
  4. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 478
  5. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 324
  6. 2023Are aligned neural networks adversarially aligned?cited by 305
  7. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 268
  8. 2023How is ChatGPT's behavior changing over time?cited by 265
  9. 2022Prompting GPT-3 To Be Reliablecited by 258
  10. 2023PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Promptscited by 218
  11. 2022On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoningcited by 191
  12. 2022MRKL Systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoningcited by 153
  13. 2022Prompting PaLM for Translation: Assessing Strategies and Performancecited by 144
  14. 2020A Survey of Privacy Attacks in Machine Learningcited by 143
  15. 2023On the Adversarial Robustness of Multi-Modal Foundation Modelscited by 142
  16. 2021Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Modelscited by 139
  17. 2020Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehensioncited by 131
  18. 2023Abusing Images and Sounds for Indirect Instruction Injection in Multi-Modal LLMscited by 117
  19. 2023AdvCLIP: Downstream-agnostic Adversarial Examples in Multimodal Contrastive Learningcited by 101
  20. 2021Prompt Waywardness: The Curious Case of Discretized Interpretation of Continuous Promptscited by 64
  21. 2023Misusing Tools in Large Language Models With Visual Adversarial Examplescited by 40
  22. 2023Adversarial Illusions in Multi-Modal Embeddingscited by 28
  23. 2023CohortGPT: An Enhanced GPT for Participant Recruitment in Clinical Studycited by 19
  24. 2023Protect Your Prompts: Protocols for IP Protection in LLM Applicationscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 27; 2025: 17; 2026: 17.

65 papers.

Sort:

By area:

  1. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 441
  2. 2024AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMscited by 160
  3. 2024EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakagecited by 146
  4. 2024Attacks, Defenses and Evaluations for LLM Conversation Safety: A Surveycited by 124
  5. 2025Large Language Models Often Know When They Are Being Evaluatedcited by 105
  6. 2023Jatmo: Prompt Injection Defense by Task-Specific Finetuningcited by 89
  7. 2024Coercing LLMs to do and reveal (almost) anythingcited by 88
  8. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Modelscited by 67
  9. 2024Introducing v0.5 of the AI Safety Benchmark from MLCommonscited by 60
  10. 2024CYBERSECEVAL 3: Advancing the Evaluation of Cybersecurity Risks and Capabilities in Large Language Modelscited by 56
  11. 2023Can LLMs Follow Simple Rules?cited by 45
  12. 2024Breaking Down the Defenses: A Comparative Survey of Attacks on Large Language Modelscited by 42
38 more
  1. 2024Jailbreaking and Mitigation of Vulnerabilities in Large Language Modelscited by 33
  2. 2024How (un)ethical are instruction-centric responses of LLMs? Unveiling the vulnerabilities of safety guardrails to harmful queriescited by 26
  3. 2025Agentic AI Security: Threats, Defenses, Evaluation, and Open Challengescited by 26
  4. 2025Simple Prompt Injection Attacks Can Leak Personal Data Observed by LLM Agents During Task Executioncited by 25
  5. 2024Diverse and Effective Red Teaming with Auto-generated Rewards and Multi-step Reinforcement Learningcited by 23
  6. 2024GoEX: Perspectives and Designs Towards a Runtime for Autonomous LLM Applicationscited by 22
  7. 2023Summon a Demon and Bind it: A Grounded Theory of LLM Red Teamingcited by 20
  8. 2024The VLLM Safety Paradox: Dual Ease in Jailbreak Attack and Defensecited by 20
  9. 2026AgentLeak: A Benchmark for Internal-Channel Privacy Leakage in Multi-Agent LLM Systemscited by 20
  10. 2025NeuroStrike: Neuron-Level Attacks on Aligned LLMscited by 18
  11. 2023Fair Abstractive Summarization of Diverse Perspectivescited by 17
  12. 2025Standard Benchmarks Fail -- Auditing LLM Agents in Finance Must Prioritize Riskcited by 13
  13. 2025Prompt Injection 2.0: Hybrid AI Threatscited by 12
  14. 2026Prompt Injection as Role Confusioncited by 11
  15. 2026OpenClaw PRISM: A Zero-Fork, Defense-in-Depth Runtime Security Layer for Tool-Augmented LLM Agentscited by 10
  16. 2025Using Mechanistic Interpretability to Craft Adversarial Attacks against Large Language Modelscited by 9
  17. 2024Recent advancements in LLM Red-Teaming: Techniques, Defenses, and Ethical Considerationscited by 8
  18. 2024LoRA-Guard: Parameter-Efficient Guardrail Adaptation for Content Moderation of Large Language Modelscited by 7
  19. 2024SMILES-Prompting: A Novel Approach to LLM Jailbreak Attacks in Chemical Synthesiscited by 7
  20. 2024Human-Interpretable Adversarial Prompt Attack on Large Language Models with Situational Contextcited by 6
  21. 2024Jailbreak Defense in a Narrow Domain: Limitations of Existing Methods and a New Transcript-Classifier Approachcited by 6
  22. 2025The Illusion of Role Separation: Hidden Shortcuts in LLM Role Learning (and How to Fix Them)cited by 6
  23. 2024The Drama Machine: Simulating Character Development with LLM Agentscited by 5
  24. 2024SoK: Prompt Hacking of Large Language Modelscited by 5
  25. 2025MOCHA: Are Code Language Models Robust Against Multi-Turn Malicious Coding Prompts?cited by 5
  26. 2026GPT-Red: Automated Red Teaming via Self-Play at Scalecited by 5
  27. 2026Visual Inception: Compromising Long-term Planning in Agentic Recommenders via Multimodal Memory Poisoningcited by 4
  28. 2024The Art of Deception: Robust Backdoor Attack using Dynamic Stacking of Triggerscited by 3
  29. 2024Human-Readable Adversarial Prompts: An Investigation into LLM Vulnerabilities Using Situational Contextcited by 3
  30. 2025Automating Prompt Leakage Attacks on Large Language Models Using Agentic Approachcited by 3
  31. 2025A Call to Action for a Secure-by-Design Generative AI Paradigmcited by 3
  32. 2025Cloud Investigation Automation Framework (CIAF): An AI-Driven Approach to Cloud Forensicscited by 3
  33. 2026Jailbreaking Vision-Language Models Through the Visual Modalitycited by 3
  34. 2024Whispers in the Machine: Confidentiality in Agentic Systemscited by 2
  35. 2025The Structural Safety Generalization Problemcited by 2
  36. 2026Guardrails for trust, safety, and ethical development and deployment of Large Language Models (LLM)cited by 2
  37. 2026When the Prompt Becomes Visual: Vision-Centric Jailbreak Attacks for Large Image Editing Modelscited by 2
  38. 2026enclawed: A Configurable, Sector-Neutral Hardening Framework for Single-User AI Assistant Gatewayscited by 2

50 most cited of 65 All 65 (JSON)