# ProxyPrompt: Securing System Prompts against Prompt Extraction Attacks

arXiv: https://arxiv.org/abs/2505.11459 · HTML: https://mlsanity.com/paper/2505.11459/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 May 2025.

## Built on by (most cited first, 5 of 5)

- [A Survey of Context Engineering for Large Language Models](https://mlsanity.com/paper/2507.13334/) (2025)
- [Just Ask: Curious Code Agents Reveal System Prompts in Frontier LLMs](https://mlsanity.com/paper/2601.21233/) (2026)
- [PSM: Prompt Sensitivity Minimization via LLM-Guided Black-Box Optimization](https://mlsanity.com/paper/2511.16209/) (2025)
- [Evaluation and Hardening of LLM System Instructions Against Extraction via Encoding Attacks](https://mlsanity.com/paper/2604.01039/) (2026)
- [ECLIPSE: Self-Evolving Stealthy Prompt Injection Attack against Long-Horizon Agentic Systems](https://mlsanity.com/paper/2608.30441/) (2026)

## Abstract (arXiv)

The integration of large language models (LLMs) into a wide range of applications has highlighted the critical role of well-crafted system prompts, which require extensive testing and domain expertise. These prompts enhance task performance but may also encode sensitive information and filtering criteria, posing security risks if exposed. Recent research shows that system prompts are vulnerable to extraction attacks, while existing defenses are either easily bypassed or require constant updates to address new threats. In this work, we introduce ProxyPrompt, a novel defense mechanism that prevents prompt leakage by replacing the original prompt with a proxy. This proxy maintains the original task's utility while obfuscating the extracted prompt, ensuring attackers cannot reproduce the task or access sensitive information. Comprehensive evaluations on 264 LLM and system prompt pairs show that ProxyPrompt protects 94.70% of prompts from extraction attacks, outperforming the next-best defense, which only achieves 42.80%.