mlsanityData to 7 Oct 2026

Prompt Injection Detection is Regime-Dependent: A Deployment-Aware Evaluation with Interpretable Structural Signals

arXiv 2605.26999 ↗First posted 26 May 2026cs.CL cs.CRcited by 0

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

Prompt injection poses a critical threat to the safe deployment of large language models, yet existing detection approaches are typically evaluated under limited settings that do not reflect real-world operating constraints. In this work, we present a deployment-aware evaluation of prompt injection detection using a multi-model and multi-regime experimental framework. We compare lexical, semantic, structural, and transformer-based detectors across multiple out-of-distribution settings, repeated data splits, and both ranking and thresholded deployment metrics. We introduce interpretable structural signals that capture hierarchy overrides, system prompt spoofing, role redefinition, and evasion patterns, and assess their contribution both within sparse models and in combination with strong encoder baselines. Our results show that detection performance is highly regime-dependent and sensitive to threshold selection, with no single model dominating across all settings. Transformer-based models achieve the strongest overall performance, while structural signals provide modest but consistent gains in certain regimes and improve low false positive rate behaviour in harder scenarios. These findings highlight the gap between ranking performance and deployment effectiveness and underscore the importance of evaluating prompt injection defences under realistic operational constraints. Code will be released.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 22 of its 37 references that are in the map. First posted on arXiv 26 May 2026.

Builds on

22 of 37 references.

Closest ancestors · 8

Same direction, newest first

  1. 2026Prompt Injection as Role Confusion22 Feb
  2. 2026When Benchmarks Lie: Evaluating Malicious Prompt Classifiers Under True Distribution Shift15 Feb
  3. 2026Prompt Injection Attacks on Agentic Coding Assistants: A Systematic Analysis of Vulnerabilities in Skills, Tools, and Protocol Ecosystems24 Jan
  4. 2025DataSentinel: A Game-Theoretic Detection of Prompt Injection Attacks15 Apr
  5. 2024InjecGuard: Benchmarking and Mitigating Over-defense in Prompt Injection Guardrail Models30 Oct
  6. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions19 Apr
  7. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models21 Dec
  8. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection23 Feb

Other references · 14

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2017On Calibration of Modern Neural Networkscited by 4,860
  3. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  4. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  5. 2017Selective Classification for Deep Neural Networkscited by 799
  6. 2020On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselinescited by 254
  7. 2025Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teamingcited by 172
  8. 2024Don't Listen To Me: Understanding and Exploring Jailbreak Prompts of Large Language Modelscited by 77
  9. 2024Prompt Obfuscation for Large Language Modelscited by 10
  10. 2024Diversity Helps Jailbreak Large Language Modelscited by 9
  11. 2025Anyone Can Jailbreak: Prompt-Based Attacks on LLMs and T2Iscited by 6
  12. 2025Revisiting the Reliability of Language Models in Instruction-Followingcited by 6
2 more
  1. 2026Low-Effort Jailbreak Attacks Against Text-to-Image Safety Filterscited by 5
  2. 2026An Empirical Study of Multi-Generation Sampling for Jailbreak Detection in Large Language Modelscited by 2

Built on by

Not cited yet.