# System Prompt Extraction Attacks and Defenses in Large Language Models

arXiv: https://arxiv.org/abs/2505.23817 · HTML: https://mlsanity.com/paper/2505.23817/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 May 2025.

## Built on by (most cited first, 6 of 6)

- [Red-Teaming Coding Agents from a Tool-Invocation Perspective: An Empirical Security Assessment](https://mlsanity.com/paper/2509.05755/) (2025)
- [Measuring Real-World Prompt Injection Attacks in LLM-based Resume Screening](https://mlsanity.com/paper/2605.28999/) (2026)
- [Multi-turn Jailbreaking Attack in Multi-Modal Large Language Models](https://mlsanity.com/paper/2601.05339/) (2026)
- [Convergence dynamics of Agent-to-Agent Interactions with Misaligned objectives](https://mlsanity.com/paper/2511.08710/) (2025)
- [Jailbreaking Large Vision Language Models in Intelligent Transportation Systems](https://mlsanity.com/paper/2511.13892/) (2025)
- [SparSEEty: Extracting Tokens from Sparsity-Exploiting LLM Serving Systems via Deterministic Side Channels](https://mlsanity.com/paper/2608.02995/) (2026)

## Abstract (arXiv)

The system prompt in Large Language Models (LLMs) plays a pivotal role in guiding model behavior and response generation. Often containing private configuration details, user roles, and operational instructions, the system prompt has become an emerging attack target. Recent studies have shown that LLM system prompts are highly susceptible to extraction attacks through meticulously designed queries, raising significant privacy and security concerns. Despite the growing threat, there is a lack of systematic studies of system prompt extraction attacks and defenses. In this paper, we present a comprehensive framework, SPE-LLM, to systematically evaluate System Prompt Extraction attacks and defenses in LLMs. First, we design a set of novel adversarial queries that effectively extract system prompts in state-of-the-art (SOTA) LLMs, demonstrating the severe risks of LLM system prompt extraction attacks. Second, we propose three defense techniques to mitigate system prompt extraction attacks in LLMs, providing practical solutions for secure LLM deployments. Third, we introduce a set of rigorous evaluation metrics to accurately quantify the severity of system prompt extraction attacks in LLMs and conduct comprehensive experiments across multiple benchmark datasets, which validates the efficacy of our proposed SPE-LLM framework.