mlsanityData to 14 Oct 2026

Neural Exec: Learning (and Learning from) Execution Triggers for Prompt Injection Attacks

arXiv 2403.03792 ↗First posted 6 Mar 2024cs.CR cs.LGcited by 85

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

We introduce a new family of prompt injection attacks, termed Neural Exec. Unlike known attacks that rely on handcrafted strings (e.g., "Ignore previous instructions and..."), we show that it is possible to conceptualize the creation of execution triggers as a differentiable search problem and use learning-based methods to autonomously generate them. Our results demonstrate that a motivated adversary can forge triggers that are not only drastically more effective than current handcrafted ones but also exhibit inherent flexibility in shape, properties, and functionality. In this direction, we show that an attacker can design and generate Neural Execs capable of persisting through multi-stage preprocessing pipelines, such as in the case of Retrieval-Augmented Generation (RAG)-based applications. More critically, our findings show that attackers can produce triggers that deviate markedly in form and shape from any known attack, sidestepping existing blacklist-based detection and sanitation approaches.

Key facts. Cited by 85 papers in the map (data through 14 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 15 of its 45 references that are in the map. First posted on arXiv 6 Mar 2024.

Builds on

15 of 45 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Prompt Injection attack against LLM-integrated Applications8 Jun
  2. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection23 Feb

Other references · 13

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2023Mistral 7Bcited by 3,164
  5. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  6. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  7. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  8. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 832
  9. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 722
  10. 2023Are aligned neural networks adversarially aligned?cited by 315
  11. 2020AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Promptscited by 249
  12. 2016Professor Forcing: A New Algorithm for Training Recurrent Networkscited by 216
1 more
  1. 2023InstructZero: Efficient Instruction Optimization for Black-Box Large Language Modelscited by 68

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 18; 2025: 39; 2026: 28.

85 papers.

Sort:

By area:

  1. 2025Defeating Prompt Injections by Designcited by 297
  2. 2024SecAlign: Defending Against Prompt Injection with Preference Optimizationcited by 210
  3. 2024AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agentscited by 145
  4. 2025DataSentinel: A Game-Theoretic Detection of Prompt Injection Attackscited by 143
  5. 2024ContextCite: Attributing Model Generation to Contextcited by 98
  6. 2024Attention Tracker: Detecting Prompt Injection Attacks in LLMscited by 95
  7. 2024Trustworthiness in Retrieval-Augmented Generation Systems: A Surveycited by 83
  8. 2024Can LLMs Separate Instructions From Data? And What Do We Even Mean By That?cited by 81
  9. 2024Get my drift? Catching LLM Task Drift with Activation Deltascited by 72
  10. 2024Imprompter: Tricking LLM Agents into Improper Tool Usecited by 68
  11. 2025DRIFT: Dynamic Rule-Based Defense with Injection Isolation for Securing LLM Agentscited by 66
  12. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agentscited by 65
38 more
  1. 2024LLMmap: Fingerprinting For Large Language Modelscited by 58
  2. 2025ACE: A Security Architecture for LLM-Integrated App Systemscited by 55
  3. 2025Multi-Agent Systems Execute Arbitrary Malicious Codecited by 53
  4. 2024Phantom: General Backdoor Attacks on Retrieval Augmented Language Generationcited by 49
  5. 2024InjecGuard: Benchmarking and Mitigating Over-defense in Prompt Injection Guardrail Modelscited by 42
  6. 2026AttriGuard: Defeating Indirect Prompt Injection in LLM Agents via Causal Attribution of Tool Invocationscited by 38
  7. 2025A Critical Evaluation of Defenses against Prompt Injection Attackscited by 36
  8. 2025RL Is a Hammer and LLMs Are Nails: A Simple Reinforcement Learning Recipe for Strong Prompt Injectioncited by 36
  9. 2025Defending Against Prompt Injection with DataFiltercited by 35
  10. 2025Defending Against Prompt Injection With a Few DefensiveTokenscited by 33
  11. 2025Security Concerns for Large Language Models: A Surveycited by 29
  12. 2025PromptLocate: Localizing Prompt Injection Attackscited by 28
  13. 2025Real AI Agents with Fake Memories: Fatal Context Manipulation Attacks on Web3 Agentscited by 24
  14. 2024Enhancing Prompt Injection Attacks to LLMs via Poisoning Alignmentcited by 21
  15. 2024GASLITEing the Retrieval: Exploring Vulnerabilities in Dense Embedding-based Searchcited by 20
  16. 2025PIShield: Detecting Prompt Injection Attacks via Intrinsic LLM Featurescited by 20
  17. 2025ObliInjection: Order-Oblivious Prompt Injection Attack to LLM Agents with Multi-source Datacited by 18
  18. 2024PROMPTFUZZ: Harnessing Fuzzing Techniques for Robust Testing of Prompt Injection in LLMscited by 17
  19. 2025Fun-tuning: Characterizing the Vulnerability of Proprietary LLMs to Optimization-based Prompt Injection Attacks via the Fine-Tuning Interfacecited by 17
  20. 2025CPA-RAG:Covert Poisoning Attacks on Retrieval-Augmented Generation in Large Language Modelscited by 17
  21. 2025ceLLMate: Sandboxing Browser AI Agentscited by 16
  22. 2026Overcoming the Retrieval Barrier: Indirect Prompt Injection in the Wild for LLM Systemscited by 16
  23. 2026SoK: The Attack Surface of Agentic AI - Tools and Autonomycited by 16
  24. 2024Hacking Back the AI-Hacker: Prompt Injection as a Defense Against LLM-driven Cyberattackscited by 15
  25. 2025Position: Adversarial ML for LLMs Is Not Making Any Progresscited by 15
  26. 2025May I have your Attention? Breaking Fine-Tuning based Prompt Injection Defenses using Architecture-Aware Attackscited by 15
  27. 2026Aligning Provenance with Authorization: A Dual-Graph Defense for LLM Agentscited by 15
  28. 2025TracLLM: A Generic Framework for Attributing Long Context LLMscited by 13
  29. 2025How Not to Detect Prompt Injections with an LLMcited by 12
  30. 2025Building Safe GenAI Applications: An End-to-End Overview of Red Teaming for Large Language Modelscited by 11
  31. 2026PIArena: A Platform for Prompt Injection Evaluationcited by 11
  32. 2025GRADA: Graph-based Reranking against Adversarial Documents Attackcited by 10
  33. 2025SecInfer: Preventing Prompt Injection via Inference-time Scalingcited by 10
  34. 2025Attention is All You Need to Defend Against Indirect Prompt Injection Attacks in LLMscited by 10
  35. 2025Stronger Enforcement of Instruction Hierarchy via Augmented Intermediate Representationscited by 9
  36. 2025Fingerprinting LLMs via Prompt Injectioncited by 8
  37. 2025Gandalf the Red: Adaptive Security for LLMscited by 7
  38. 2025AttnTrace: Contextual Attribution of Prompt Injection and Knowledge Corruptioncited by 7

50 most cited of 85 All 85 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Neural Architectural Backdoors
  2. 2025Defending against Indirect Prompt Injection by Instruction Detection
  3. 2025Exploiting Web Search Tools of AI Agents for Data Exfiltration
  4. 2026Bypassing Prompt Injection Detectors through Evasive Injections
  5. 2025Cybersecurity AI: Hacking the AI Hackers via Prompt Injection
  6. 2026Defense Against Indirect Prompt Injection via Tool Result Parsing
  7. 2026AlignSentinel: Alignment-Aware Detection of Prompt Injection Attacks
  8. 2025Securing AI Agents Against Prompt Injection Attacks