PromptCOS: Towards Content-only System Prompt Copyright Auditing for LLMs
On the map: LLM safety and security → Watermarking generated content and models
Abstract (arXiv)
System prompts are critical for shaping the behavior and output quality of large language model (LLM)-based applications, driving substantial investment in optimizing high-quality prompts beyond traditional handcrafted designs. However, as system prompts become valuable intellectual property, they are increasingly vulnerable to prompt theft and unauthorized use, highlighting the urgent need for effective copyright auditing, especially watermarking. Existing methods rely on verifying subtle logit distribution shifts triggered by a query. We observe that this logit-dependent verification framework is impractical in real-world content-only settings, primarily because (1) random sampling makes content-level generation unstable for verification, and (2) stronger instructions needed for content-level signals compromise prompt fidelity. To overcome these challenges, we propose PromptCOS, the first content-only system prompt copyright auditing method based on content-level output similarity. PromptCOS achieves watermark stability by designing a cyclic output signal as the conditional instruction's target. It preserves prompt fidelity by injecting a small set of auxiliary tokens to encode the watermark, leaving the main prompt untouched. Furthermore, to ensure robustness against malicious removal, we optimize cover tokens, i.e., critical tokens in the original prompt, to ensure that removing auxiliary tokens causes severe performance degradation. Experimental results show that promptCOS achieves high effectiveness (99.3% average watermark similarity), strong distinctiveness (60.8% higher than the best baseline), high fidelity (accuracy degradation no greater than 0.6%), robustness (resilience against four potential attack categories), and high computational efficiency (up to 98.1% cost saving).
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Watermarking generated content and models, one of 12 directions in LLM safety and security. Builds on 52 of its 75 references that are in the map. First posted on arXiv 3 Sep 2025.
Builds on
52 of 75 references.
Closest ancestors · 9
Same direction, newest first
- 2024SoK: On the Role and Future of AIGC Watermarking in the Era of Gen-AI18 Nov
- 2024Safe-SD: Safe and Traceable Stable Diffusion with Text Prompt Trigger for Invisible Generative Watermarking18 Jul
- 2024Explanation as a Watermark: Towards Harmless and Multi-bit Model Ownership Verification via Watermarking Feature Attribution8 May
- 2023A Survey of Text Watermarking in the Era of Large Language Models13 Dec
- 2023REMARK-LLM: A Robust and Efficient Watermarking Framework for Generative Large Language Models18 Oct
- 2023Robust Distortion-free Watermarks for Language Models28 Jul
- 2023Provable Robust Watermarking for AI-Generated Text30 Jun
- 2023A Watermark for Large Language Models24 Jan
- 2021Tracing Text Provenance via Context-Aware Lexical Substitution15 Dec
Other references · 43
- 2017Attention Is All You Needcited by 60,361
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
- 2021Training Verifiers to Solve Math Word Problemscited by 10,109
- 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
- 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
- 2023Code Llama: Open Foundation Models for Codecited by 2,139
- 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
- 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
- 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
- 2019How Can We Know What Language Models Know?cited by 959
- 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
31 more
- 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 688
- 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 399
- 2023Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discoverycited by 336
- 2023Mathematical Capabilities of ChatGPTcited by 268
- 2020AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Promptscited by 249
- 2024CodeGemma: Open Code Models Based on Gemmacited by 150
- 2024PLeak: Prompt Leaking Attacks against Large Language Model Applicationscited by 127
- 2024The Prompt Report: A Systematic Survey of Prompt Engineering Techniquescited by 121
- 2024Mitigating Catastrophic Forgetting in Large Language Models with Self-Synthesized Rehearsalcited by 118
- 2024Parrot: Efficient Serving of LLM-based Applications with Semantic Variablecited by 99
- 2022Learning to Break the Loop: Analyzing and Mitigating Repetitions for Neural Text Generationcited by 94
- 2021On Transferability of Prompt Tuning for Natural Language Processingcited by 75
- 2024Instructional Fingerprinting of Large Language Modelscited by 68
- 2023Prompt Stealing Attacks Against Text-to-Image Generation Modelscited by 51
- 2024PARDEN, Can You Repeat That? Defending against Jailbreaks via Repetitioncited by 35
- 2022Why So Toxic? Measuring and Triggering Toxic Behavior in Open-Domain Chatbotscited by 33
- 2023PromptCARE: Prompt Copyright Protection by Watermark Injection and Verificationcited by 32
- 2024Teach LLMs to Phish: Stealing Private Information from Language Modelscited by 32
- 2023AutoHint: Automatic Prompt Optimization with Hint Generationcited by 30
- 2024PEFTGuard: Detecting Backdoor Attacks Against Parameter-Efficient Fine-Tuningcited by 26
- 2019FAQ Retrieval using Query-Question Similarity and BERT-Based Query-Answer Relevancecited by 25
- 2024Citation-Enhanced Generation for LLM-based Chatbotscited by 25
- 2024Is Difficulty Calibration All We Need? Towards More Practical Membership Inference Attackscited by 21
- 2024PromSec: Prompt Optimization for Secure Generation of Functional Source Code with Large Language Models (LLMs)cited by 19
- 2025Probe before You Talk: Towards Black-box Defense against Backdoor Unalignment for Large Language Modelscited by 19
- 2025Rethinking Data Protection in the (Generative) Artificial Intelligence Eracited by 18
- 2024On the (In)Security of LLM App Storescited by 14
- 2025On Benchmarking Code LLMs for Android Malware Analysiscited by 13
- 2023Stateful Defenses for Machine Learning Models Are Not Yet Secure Against Black-box Attackscited by 11
- 2024PRSA: Prompt Stealing Attacks against Real-World Prompt Servicescited by 10
- 2024CoSearchAgent: A Lightweight Collaborative Search Agent with Large Language Modelscited by 7
Built on by
Citing papers per year: 2026: 3.
3 papers.
- 2026Echoes within the Reasoning: Stealthy and Effective Watermarking via Chain of Thoughtcited by 2
- 2026LoRA-Key: User-Centric LoRA Watermarking for Text-to-Image Diffusion Modelscited by 1
- 2026PragLocker: Protecting Agent Intellectual Property in Untrusted Deployments via Non-Portable Promptscited by 0
- 2026LoRA-Key: User-Centric LoRA Watermarking for Text-to-Image Diffusion Models28 May 2026
- 2026Echoes within the Reasoning: Stealthy and Effective Watermarking via Chain of Thought27 May 2026
- 2026PragLocker: Protecting Agent Intellectual Property in Untrusted Deployments via Non-Portable Prompts7 May 2026
Similar, not cited
Close in wording, no citation either way
- 2024Watermarking Language Models through Language Models
- 2025SWAP: Towards Copyright Auditing of Soft Prompts via Sequential Watermarking
- 2025SoK: Are Watermarks in LLMs Ready for Deployment?
- 2026PromptMark: A Prompt-Guided Iterative-Feedback Framework for Source Code Watermarking ↗
- 2024Can Watermarked LLMs be Identified by Users via Crafted Prompts?
- 2025Copyright Protection for Large Language Models: A Survey of Methods, Challenges, and Trends
- 2024Can Watermarking Large Language Models Prevent Copyrighted Text Generation and Hide Training Data?
- 2024Lost in Overlap: Exploring Logit-based Watermark Collision in LLMs