mlsanityData to 14 Oct 2026

Automatic and Universal Prompt Injection Attacks against Large Language Models

arXiv 2403.04957 ↗First posted 7 Mar 2024cs.AIcited by 119

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

Large Language Models (LLMs) excel in processing and generating human language, powered by their ability to interpret and follow instructions. However, their capabilities can be exploited through prompt injection attacks. These attacks manipulate LLM-integrated applications into producing responses aligned with the attacker's injected content, deviating from the user's actual requests. The substantial risks posed by these attacks underscore the need for a thorough understanding of the threats. Yet, research in this area faces challenges due to the lack of a unified goal for such attacks and their reliance on manually crafted prompts, complicating comprehensive assessments of prompt injection robustness. We introduce a unified framework for understanding the objectives of prompt injection attacks and present an automated gradient-based method for generating highly effective and universal prompt injection data, even in the face of defensive measures. With only five training samples (0.3% relative to the test data), our attack can achieve superior performance compared with baselines. Our findings emphasize the importance of gradient-based testing, which can avoid overestimation of robustness, especially for defense mechanisms.

Key facts. Cited by 119 papers in the map (data through 14 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 31 of its 47 references that are in the map. First posted on arXiv 7 Mar 2024.

Builds on

31 of 47 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024A Novel Evaluation Framework for Assessing Resilience Against Prompt Injection Attacks in Large Language Models2 Jan
  2. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models21 Dec
  3. 2023Maatphor: Automated Variant Analysis for Prompt Injection Attacks12 Dec
  4. 2023Assessing Prompt Injection Risks in 200+ Custom GPTs20 Nov
  5. 2023Tensor Trust: Interpretable Prompt Injection Attacks from an Online Game2 Nov
  6. 2023Formalizing and Benchmarking Prompt Injection Attacks and Defenses19 Oct
  7. 2023From Prompt Injections to SQL Injection Attacks: How Protected is Your LLM-Integrated Web Application?3 Aug
  8. 2023Safeguarding Crowdsourcing Surveys from ChatGPT with Prompt Injection15 Jun
  9. 2023Prompt Injection attack against LLM-integrated Applications8 Jun
  10. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection23 Feb

Other references · 21

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  6. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  7. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  8. 2018Neural Network Acceptability Judgmentscited by 958
  9. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 832
  10. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 785
  11. 2017Automated Hate Speech Detection and the Problem of Offensive Languagecited by 736
  12. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 651
9 more
  1. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 451
  2. 2023Detecting Language Model Attacks with Perplexitycited by 431
  3. 2015A Neural Attention Model for Abstractive Sentence Summarizationcited by 396
  4. 2023Low-Resource Languages Jailbreak GPT-4cited by 309
  5. 2024TrustLLM: Trustworthiness in Large Language Modelscited by 285
  6. 2018ShapeShifter: Robust Physical Adversarial Attack on Faster R-CNN Object Detectorcited by 159
  7. 2023Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injectioncited by 157
  8. 2017JFLEG: A Fluency Corpus and Benchmark for Grammatical Error Correctioncited by 97
  9. 2022Evaluating the Susceptibility of Pre-Trained Language Models via Handcrafted Adversarial Examplescited by 69

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 16; 2025: 65; 2026: 38.

119 papers.

Sort:

By area:

  1. 2025DataSentinel: A Game-Theoretic Detection of Prompt Injection Attackscited by 143
  2. 2024Prompt Infection: LLM-to-LLM Prompt Injection within Multi-Agent Systemscited by 133
  3. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  4. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 116
  5. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 67
  6. 2025DRIFT: Dynamic Rule-Based Defense with Injection Isolation for Securing LLM Agentscited by 66
  7. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agentscited by 65
  8. 2025A Survey of LLM-Driven AI Agent Communication: Protocols, Security Risks, and Defense Countermeasurescited by 57
  9. 2025Manipulating Multimodal Agents via Cross-Modal Prompt Injectioncited by 44
  10. 2026AgentLAB: Benchmarking LLM Agents against Long-Horizon Attackscited by 39
  11. 2024Feedback-Guided Extraction of Knowledge Base from Retrieval-Augmented LLM Applicationscited by 37
  12. 2025A Critical Evaluation of Defenses against Prompt Injection Attackscited by 36
38 more
  1. 2025RL Is a Hammer and LLMs Are Nails: A Simple Reinforcement Learning Recipe for Strong Prompt Injectioncited by 36
  2. 2025Defending Against Prompt Injection with DataFiltercited by 35
  3. 2025Red Teaming the Mind of the Machine: A Systematic Evaluation of Prompt Injection and Jailbreak Vulnerabilities in LLMscited by 34
  4. 2026The Landscape of Prompt Injection Threats in LLM Agents: From Taxonomy to Analysiscited by 30
  5. 2025LLM Agents Should Employ Security Principlescited by 29
  6. 2025PromptLocate: Localizing Prompt Injection Attackscited by 28
  7. 2025Agentic AI Security: Threats, Defenses, Evaluation, and Open Challengescited by 28
  8. 2024Securing Large Language Models: Addressing Bias, Misinformation, and Prompt Attackscited by 26
  9. 2025BrowseSafe: Understanding and Preventing Prompt Injection Within AI Browser Agentscited by 24
  10. 2024Enhancing Prompt Injection Attacks to LLMs via Poisoning Alignmentcited by 21
  11. 2026MAGIC: A Co-Evolving Attacker-Defender Adversarial Game for Robust LLM Safetycited by 21
  12. 2025PIShield: Detecting Prompt Injection Attacks via Intrinsic LLM Featurescited by 20
  13. 2024On Large Language Models in National Security Applicationscited by 19
  14. 2025Oyster-I: Beyond Refusal -- Constructive Safety Alignment for Responsible Language Modelscited by 19
  15. 2025Automatic Red Teaming LLM-based Agents with Model Context Protocol Toolscited by 18
  16. 2025ObliInjection: Order-Oblivious Prompt Injection Attack to LLM Agents with Multi-source Datacited by 18
  17. 2024PROMPTFUZZ: Harnessing Fuzzing Techniques for Robust Testing of Prompt Injection in LLMscited by 17
  18. 2024Efficient Universal Goal Hijacking with Semantics-guided Prompt Organizationcited by 16
  19. 2025Unsafe LLM-Based Search: Quantitative Analysis and Mitigation of Safety Risks in AI Web Searchcited by 16
  20. 2025Robustness via Referencing: Defending against Prompt Injection Attacks by Referencing the Executed Instructioncited by 16
  21. 2026A Framework for Formalizing LLM Agent Securitycited by 16
  22. 2025A Survey of Attacks on Large Language Modelscited by 15
  23. 2025May I have your Attention? Breaking Fine-Tuning based Prompt Injection Defenses using Architecture-Aware Attackscited by 15
  24. 2026Aligning Provenance with Authorization: A Dual-Graph Defense for LLM Agentscited by 15
  25. 2025Refining Input Guardrails: Enhancing LLM-as-a-Judge Efficiency Through Chain-of-Thought Fine-Tuning and Alignmentcited by 14
  26. 2024SafeEmbodAI: a Safety Framework for Mobile Robots in Embodied AI Systemscited by 13
  27. 2025Prompt Injection 2.0: Hybrid AI Threatscited by 13
  28. 2025How Not to Detect Prompt Injections with an LLMcited by 12
  29. 2025Robust Answers, Fragile Logic: Probing the Decoupling Hypothesis in LLM Reasoningcited by 11
  30. 2025CHAI: Command Hijacking against embodied AIcited by 11
  31. 2025Investigating the Vulnerability of LLM-as-a-Judge Architectures to Prompt-Injection Attackscited by 10
  32. 2025In-Context Watermarks for Large Language Modelscited by 10
  33. 2025TAI3: Testing Agent Integrity in Interpreting User Intentcited by 9
  34. 2025Taxonomy, Evaluation and Exploitation of IPI-Centric LLM Agent Defense Frameworkscited by 9
  35. 2025LLM Security: Vulnerabilities, Attacks, Defenses, and Countermeasurescited by 8
  36. 2026SoK: Security of Autonomous LLM Agents in Agentic Commercecited by 8
  37. 2025Gandalf the Red: Adaptive Security for LLMscited by 7
  38. 2025Bridging AI and Software Security: A Comparative Vulnerability Assessment of LLM Agent Deployment Paradigmscited by 7

50 most cited of 119 All 119 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Evaluating the Instruction-Following Robustness of Large Language Models to Prompt Injection
  2. 2025Prompt Injection as an Emerging Threat: Evaluating the Resilience of Large Language Models
  3. 2024An Early Categorization of Prompt Injection Attacks on Large Language Models
  4. 2024Defense Against Prompt Injection Attack by Leveraging Attack Techniques
  5. 2026AttackEval: A Systematic Empirical Study of Prompt Injection Attack Effectiveness Against Large Language Models
  6. 2024Scaling Behavior of Machine Translation with Large Language Models under Prompt Injection Attacks
  7. 2025System Prompt Poisoning: Persistent Attacks on Large Language Models Beyond User Injection
  8. 2025Attack and defense techniques in large language models: A survey and new perspectives