mlsanityData to 14 Oct 2026

Ignore Previous Prompt: Attack Techniques For Language Models

arXiv 2211.09527 ↗First posted 17 Nov 2022cs.CL cs.AIcited by 832

Not on the map. Why?

Cited by 403 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Transformer-based large language models (LLMs) provide a powerful foundation for natural language tasks in large-scale customer-facing applications. However, studies that explore their vulnerabilities emerging from malicious user interaction are scarce. By proposing PromptInject, a prosaic alignment framework for mask-based iterative adversarial prompt composition, we examine how GPT-3, the most widely deployed language model in production, can be easily misaligned by simple handcrafted inputs. In particular, we investigate two types of attacks -- goal hijacking and prompt leaking -- and demonstrate that even low-aptitude, but sufficiently ill-intentioned agents, can easily exploit GPT-3's stochastic nature, creating long-tail risks. The code for PromptInject is available at https://github.com/agencyenterprise/PromptInject.

Key facts. Cited by 832 papers in the map (data through 14 Oct 2026). Builds on 19 of its 44 references that are in the map. First posted on arXiv 17 Nov 2022.

Builds on

19 of 44 references.

References · 19

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  6. 2020Extracting Training Data from Large Language Modelscited by 2,189
  7. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  8. 2021Ethical and social risks of harm from Language Modelscited by 966
  9. 2021Persistent Anti-Muslim Bias in Large Language Modelscited by 377
  10. 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 356
  11. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 258
  12. 2022GrIPS: Gradient-free, Edit-based Instruction Search for Prompting Large Language Modelscited by 127
7 more
  1. 2020OpenAttack: An Open-source Textual Adversarial Attack Toolkitcited by 84
  2. 2020The Radicalization Risks of GPT-3 and Advanced Neural Language Modelscited by 76
  3. 2022Correcting Robot Plans with Natural Language Feedbackcited by 69
  4. 2022Evaluating the Susceptibility of Pre-Trained Language Models via Handcrafted Adversarial Examplescited by 69
  5. 2022X-Risk Analysis for AI Researchcited by 60
  6. 2022Identifying Adversarial Attacks on Text Classifierscited by 4
  7. 2022A Case Report On The "A.I. Locked-In Problem": social concerns with modern NLPcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 87; 2024: 174; 2025: 208; 2026: 363.

832 papers.

Sort:

By area:

  1. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
  2. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  3. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 651
  4. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 623
  5. 2023Prompt Injection attack against LLM-integrated Applicationscited by 605
  6. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 560
  7. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agentscited by 538
  8. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 493
  9. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 489
  10. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  11. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Ciphercited by 423
  12. 2024StruQ: Defending Against Prompt Injection with Structured Queriescited by 375
38 more
  1. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agentscited by 369
  2. 2023Formalizing and Benchmarking Prompt Injection Attacks and Defensescited by 364
  3. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 333
  4. 2023Tool Learning with Foundation Modelscited by 302
  5. 2025Defeating Prompt Injections by Designcited by 297
  6. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 278
  7. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Modelscited by 273
  8. 2024PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Modelscited by 267
  9. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Modelscited by 264
  10. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 263
  11. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 260
  12. 2023A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easilycited by 259
  13. 2023Challenges and Applications of Large Language Modelscited by 254
  14. 2023PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Promptscited by 226
  15. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 212
  16. 2024SecAlign: Defending Against Prompt Injection with Preference Optimizationcited by 210
  17. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 186
  18. 2023MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbotscited by 183
  19. 2023Do LLMs exhibit human-like response biases? A case study in survey designcited by 180
  20. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllabilitycited by 175
  21. 2023Image Hijacks: Adversarial Images can Control Generative Models at Runtimecited by 172
  22. 2025Prompt Injection Attack to Tool Selection in LLM Agentscited by 158
  23. 2023Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injectioncited by 157
  24. 2024AdvPrompter: Fast Adaptive Adversarial Prompting for LLMscited by 156
  25. 2024EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakagecited by 150
  26. 2023Tensor Trust: Interpretable Prompt Injection Attacks from an Online Gamecited by 145
  27. 2024AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agentscited by 145
  28. 2024Optimization-based Prompt Injection Attack to LLM-as-a-Judgecited by 143
  29. 2025DataSentinel: A Game-Theoretic Detection of Prompt Injection Attackscited by 143
  30. 2023Inspecting and Editing Knowledge Representations in Language Modelscited by 141
  31. 2024Prompt Infection: LLM-to-LLM Prompt Injection within Multi-Agent Systemscited by 133
  32. 2024Attacks, Defenses and Evaluations for LLM Conversation Safety: A Surveycited by 130
  33. 2025The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injectionscited by 129
  34. 2024PLeak: Prompt Leaking Attacks against Large Language Model Applicationscited by 127
  35. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 127
  36. 2024Automatic and Universal Prompt Injection Attacks against Large Language Modelscited by 119
  37. 2024MLLM-Protector: Ensuring MLLM's Safety without Hurting Performancecited by 117
  38. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 116

50 most cited of 832 All 832 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Breaking to Build: A Threat Model of Prompt-Based Attacks for Securing LLMs
  2. 2025Prompt Injection as an Emerging Threat: Evaluating the Resilience of Large Language Models
  3. 2026RECAP: A Resource-Efficient Method for Adversarial Prompting in Large Language Models
  4. 2025A methodological analysis of prompt perturbations and their effect on attack success rates
  5. 2026Adversarial Reframing: A Framework for Targeted Generation in Language Models
  6. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks
  7. 2026When Prompt Optimization Becomes Jailbreaking: Adaptive Red-Teaming of Large Language Models
  8. 2026An Empirical Evaluation of Prompt Injection Vulnerabilities in Large Language Models Across Multilingual and Obfuscated Attack Scenarios