mlsanityData to 14 Oct 2026

The Defense Trilemma: Why Prompt Injection Defense Wrappers Fail?

arXiv 2604.06436 ↗First posted 7 Apr 2026cs.CR cs.AIcited by 2

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

We prove that no continuous, utility-preserving wrapper defense-a function D: X→ X that preprocesses inputs before the model sees them-can make all outputs strictly safe for a language model with connected prompt space, and we characterize exactly where every such defense must fail. We establish three results under successively stronger hypotheses: boundary fixation-the defense must leave some threshold-level inputs unchanged; an ε-robust constraint-under Lipschitz regularity, a positive-measure band around fixed boundary points remains near-threshold; and a persistent unsafe region under a transversality condition, a positive-measure subset of inputs remains strictly unsafe. These constitute a defense trilemma: continuity, utility preservation, and completeness cannot coexist. We prove parallel discrete results requiring no topology, and extend to multi-turn interactions, stochastic defenses, and capacity-parity settings. The results do not preclude training-time alignment, architectural changes, or defenses that sacrifice utility. The full theory is mechanically verified in Lean 4 and validated empirically on three LLMs.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 26 of its 41 references that are in the map. First posted on arXiv 7 Apr 2026.

Builds on

26 of 41 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents3 Oct
  2. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts26 Feb
  3. 2023MART: Improving LLM Safety with Multi-round Automatic Red-Teaming13 Nov
  4. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection23 Feb

Other references · 22

  1. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  2. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  3. 2013Intriguing properties of neural networkscited by 5,155
  4. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  5. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  6. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
  7. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  8. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,340
  9. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  10. 2018Robustness May Be at Odds with Accuracycited by 880
  11. 2017Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networkscited by 744
  12. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 672
10 more
  1. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 529
  2. 2023Detecting Language Model Attacks with Perplexitycited by 431
  3. 2015Illuminating search spaces by mapping elitescited by 425
  4. 2016Safety Verification of Deep Neural Networkscited by 263
  5. 2020Topology of deep neural networkscited by 112
  6. 2024Weak-to-Strong Jailbreaking on Large Language Modelscited by 107
  7. 2018Adversarial vulnerability for any classifiercited by 78
  8. 2024Model Merging and Safety Alignment: One Bad Model Spoils the Bunchcited by 43
  9. 2025What Really Matters in Many-Shot Attacks? An Empirical Study of Long-Context Vulnerabilities in LLMscited by 8
  10. 2026Manifold of Failure: Behavioral Attraction Basins in Language Modelscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Safeguards Based on Copyable Context Cannot Provide Reliable Safety for LLMscited by 1
  2. 2026Towards a Data-Parameter Correspondence for LLMs: A Preliminary Discussioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Closing the Activation-Cone Blind Spot: Response-Time Probing and Unified Defense
  2. 2025A Critical Evaluation of Defenses against Prompt Injection Attacks
  3. 2025Defending Against Prompt Injection With a Few DefensiveTokens
  4. 2026Cross-Session Decomposition Attacks: Scaling Risk and Intent-Aligned Retrieval Defense
  5. 2026When LLM Defenses Backfire: Characterizing Safety, Performance, and Cost Trade-offs
  6. 2025Preventing Prompt Injection with Type-Directed Privilege Separation
  7. 2026LTBD: Learnable Trust-Boundary Delimiters for Prompt Injection Defense
  8. 2026Safety Paradox: How Enhanced Safety Awareness Leaves LLMs Vulnerable to Posterior Attack