# System Prompt Poisoning: Persistent Attacks on Large Language Models Beyond User Injection arXiv: https://arxiv.org/abs/2505.06493 ยท HTML: https://mlsanity.com/paper/2505.06493/ ## Key facts - Cited by 14 papers in the map (data through 14 Oct 2026). - Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. - Builds on 16 of its 26 references that are in the map. - First posted on arXiv 10 May 2025. ## Placed in - LLM safety and security / [Prompt injection and agent security](https://mlsanity.com/safety/prompt-injection-and-agent-security/) ## Builds on (most cited first, 16 of 16) - [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021) - [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021) - [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024) - [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023) - [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://mlsanity.com/paper/2302.12173/) (2023) - [Ignore Previous Prompt: Attack Techniques For Language Models](https://mlsanity.com/paper/2211.09527/) (2022) - [Baseline Defenses for Adversarial Attacks Against Aligned Language Models](https://mlsanity.com/paper/2309.00614/) (2023) - [InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents](https://mlsanity.com/paper/2403.02691/) (2024) - [PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models](https://mlsanity.com/paper/2402.07867/) (2024) - [Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models](https://mlsanity.com/paper/2312.14197/) (2023) - [Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks](https://mlsanity.com/paper/2310.10844/) (2023) - [Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents](https://mlsanity.com/paper/2503.00061/) (2025) - [LLM Platform Security: Applying a Systematic Evaluation Framework to OpenAI's ChatGPT Plugins](https://mlsanity.com/paper/2309.10254/) (2023) - [PoisonBench: Assessing Large Language Model Vulnerability to Data Poisoning](https://mlsanity.com/paper/2410.08811/) (2024) - [FATH: Authentication-based Test-time Defense against Indirect Prompt Injection Attacks](https://mlsanity.com/paper/2410.21492/) (2024) - [On the (In)Security of LLM App Stores](https://mlsanity.com/paper/2407.08422/) (2024) ## Built on by (most cited first, 14 of 14) - [Memory Poisoning Attack and Defense on Memory Based LLM-Agents](https://mlsanity.com/paper/2601.05504/) (2026) - [Security Concerns for Large Language Models: A Survey](https://mlsanity.com/paper/2505.18889/) (2025) - [Agentic AI Security: Threats, Defenses, Evaluation, and Open Challenges](https://mlsanity.com/paper/2510.23883/) (2025) - [A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Models](https://mlsanity.com/paper/2509.03871/) (2025) - [ETDI: Mitigating Tool Squatting and Rug Pull Attacks in Model Context Protocol (MCP) by using OAuth-Enhanced Tool Definitions and Policy-Based Access Control](https://mlsanity.com/paper/2506.01333/) (2025) - [Logic layer Prompt Control Injection (LPCI): A Novel Security Vulnerability Class in Agentic Systems](https://mlsanity.com/paper/2507.10457/) (2025) - [Eliciting and Analyzing Emergent Misalignment in State-of-the-Art Large Language Models](https://mlsanity.com/paper/2508.04196/) (2025) - [Multimodal Prompt Injection Attacks: Risks and Defenses for Modern LLMs](https://mlsanity.com/paper/2509.05883/) (2025) - [FuncPoison: Poisoning Function Library to Hijack Multi-agent Autonomous Driving Systems](https://mlsanity.com/paper/2509.24408/) (2025) - [Rethinking Reasoning: A Survey on Reasoning-based Backdoors in LLMs](https://mlsanity.com/paper/2510.07697/) (2025) - [Inference-Time Backdoors via Chat Templates: From LLM Supply Chains to Agentic System Compromise](https://mlsanity.com/paper/2602.04653/) (2026) - [The Surface You Test Is Not the Surface That Breaks](https://mlsanity.com/paper/2605.30454/) (2026) - [ENDOPROMPT: Victim-Side Pseudo-References for Utility Degradation](https://mlsanity.com/paper/2609.29948/) (2026) - [The System Prompt Illusion: How Instruction Preambles Modify Computation in Language Models](https://mlsanity.com/paper/2609.38205/) (2026) ## Abstract (arXiv) Large language models (LLMs) have gained widespread adoption across diverse applications due to their impressive generative capabilities. Their plug-and-play nature enables both developers and end users to interact with these models through simple prompts. However, as LLMs become more integrated into various systems in diverse domains, concerns around their security are growing. Existing studies mainly focus on threats arising from user prompts (e.g. prompt injection attack) and model output (e.g. model inversion attack), while the security of system prompts remains largely overlooked. This work bridges the critical gap. We introduce system prompt poisoning, a new attack vector against LLMs that, unlike traditional user prompt injection, poisons system prompts hence persistently impacts all subsequent user interactions and model responses. We systematically investigate four practical attack strategies in various poisoning scenarios. Through demonstration on both generative and reasoning LLMs, we show that system prompt poisoning is highly feasible without requiring jailbreak techniques, and effective across a wide range of tasks, including those in mathematics, coding, logical reasoning, and natural language processing. Importantly, our findings reveal that the attack remains effective even when user prompts employ advanced prompting techniques like chain-of-thought (CoT). We also show that such techniques, including CoT and retrieval-augmentation-generation (RAG), which are proven to be effective for improving LLM performance in a wide range of tasks, are significantly weakened in their effectiveness by system prompt poisoning.