mlsanityData to 14 Oct 2026

The System Prompt Is the Attack Surface: How LLM Agent Configuration Shapes Security and Creates Exploitable Vulnerabilities

arXiv 2603.25056 ↗First posted 26 Mar 2026cs.CR cs.AIcited by 2

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

System prompt configuration can make the difference between near-total phishing blindness and near-perfect detection in LLM email agents. We present PhishNChips, a study of 11 models under 10 prompt strategies, showing that prompt-model interaction is a first-order security variable: a single model's phishing bypass rate ranges from under 1% to 97% depending on how it is configured, while the false-positive cost of the same prompt varies sharply across models. We then show that optimizing prompts around highly predictive signals can improve benchmark performance, reaching up to 93.7% recall at 3.8% false positive rate, but also creates a brittle attack surface. In particular, domain-matching strategies perform well when legitimate emails mostly have matched sender and URL domains, yet degrade sharply when attackers invert that signal by registering matching infrastructure. Response-trace analysis shows that 98% of successful bypasses reason in ways consistent with the inverted signal: the models are following the instruction, but the instruction's core assumption has become false. A counter-intuitive corollary follows: making prompts more specific can degrade already-capable models by replacing broader multi-signal reasoning with exploitable single-signal dependence. We characterize the resulting tension between detection, usability, and adversarial robustness as a navigable tradeoff, introduce Safetility, a deployability-aware metric that penalizes false positives, and argue that closing the adversarial gap likely requires tool augmentation with external ground truth.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 10 of its 17 references that are in the map. First posted on arXiv 26 Mar 2026.

Builds on

10 of 17 references.

References · 10

  1. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  2. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
  3. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  4. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 536
  5. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 260
  6. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 230
  7. 2025Fraud-R1 : A Multi-Round Benchmark for Assessing the Robustness of LLM Against Augmented Fraud and Phishing Inducementscited by 32
  8. 2025Security Concerns for Large Language Models: A Surveycited by 29
  9. 2024DetoxBench: Benchmarking Large Language Models for Multitask Fraud & Abuse Detectioncited by 5
  10. 2024SpearBot: Leveraging Large Language Models in a Generative-Critique Framework for Spear-Phishing Email Generationcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2025Your Mailbox Is Mine: Prompt Injection Attacks Against Real-World LLM Email Agents
  2. 2025LLMail-Inject: A Dataset from a Realistic Adaptive Prompt Injection Challenge
  3. 2025MultiPhishGuard: An Explainable and Adaptive Multi-Agent LLM System for Phishing Email Detection
  4. 2025Efficient but Vulnerable: Benchmarking and Defending LLM Batch Prompting Attack
  5. 2026An Empirical Evaluation of Prompt Injection Vulnerabilities in Large Language Models Across Multilingual and Obfuscated Attack Scenarios
  6. 2026Clouding the Mirror: Stealthy Prompt Injection Attacks Targeting LLM-based Phishing Detection
  7. 2025Automating Prompt Leakage Attacks on Large Language Models Using Agentic Approach
  8. 2025Multi-Stage Prompt Inference Attacks on Enterprise LLM Systems