mlsanityData to 14 Oct 2026

PromptCOS: Towards Content-only System Prompt Copyright Auditing for LLMs

arXiv 2509.03117 ↗First posted 3 Sep 2025cs.CRcited by 3

On the map: LLM safety and security → Watermarking generated content and models

Abstract (arXiv)

System prompts are critical for shaping the behavior and output quality of large language model (LLM)-based applications, driving substantial investment in optimizing high-quality prompts beyond traditional handcrafted designs. However, as system prompts become valuable intellectual property, they are increasingly vulnerable to prompt theft and unauthorized use, highlighting the urgent need for effective copyright auditing, especially watermarking. Existing methods rely on verifying subtle logit distribution shifts triggered by a query. We observe that this logit-dependent verification framework is impractical in real-world content-only settings, primarily because (1) random sampling makes content-level generation unstable for verification, and (2) stronger instructions needed for content-level signals compromise prompt fidelity. To overcome these challenges, we propose PromptCOS, the first content-only system prompt copyright auditing method based on content-level output similarity. PromptCOS achieves watermark stability by designing a cyclic output signal as the conditional instruction's target. It preserves prompt fidelity by injecting a small set of auxiliary tokens to encode the watermark, leaving the main prompt untouched. Furthermore, to ensure robustness against malicious removal, we optimize cover tokens, i.e., critical tokens in the original prompt, to ensure that removing auxiliary tokens causes severe performance degradation. Experimental results show that promptCOS achieves high effectiveness (99.3% average watermark similarity), strong distinctiveness (60.8% higher than the best baseline), high fidelity (accuracy degradation no greater than 0.6%), robustness (resilience against four potential attack categories), and high computational efficiency (up to 98.1% cost saving).

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Watermarking generated content and models, one of 12 directions in LLM safety and security. Builds on 52 of its 75 references that are in the map. First posted on arXiv 3 Sep 2025.

Builds on

52 of 75 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024SoK: On the Role and Future of AIGC Watermarking in the Era of Gen-AI18 Nov
  2. 2024Safe-SD: Safe and Traceable Stable Diffusion with Text Prompt Trigger for Invisible Generative Watermarking18 Jul
  3. 2024Explanation as a Watermark: Towards Harmless and Multi-bit Model Ownership Verification via Watermarking Feature Attribution8 May
  4. 2023A Survey of Text Watermarking in the Era of Large Language Models13 Dec
  5. 2023REMARK-LLM: A Robust and Efficient Watermarking Framework for Generative Large Language Models18 Oct
  6. 2023Robust Distortion-free Watermarks for Language Models28 Jul
  7. 2023Provable Robust Watermarking for AI-Generated Text30 Jun
  8. 2023A Watermark for Large Language Models24 Jan
  9. 2021Tracing Text Provenance via Context-Aware Lexical Substitution15 Dec

Other references · 43

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  6. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  7. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  8. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  9. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  10. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
  11. 2019How Can We Know What Language Models Know?cited by 959
  12. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
31 more
  1. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 688
  2. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 399
  3. 2023Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discoverycited by 336
  4. 2023Mathematical Capabilities of ChatGPTcited by 268
  5. 2020AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Promptscited by 249
  6. 2024CodeGemma: Open Code Models Based on Gemmacited by 150
  7. 2024PLeak: Prompt Leaking Attacks against Large Language Model Applicationscited by 127
  8. 2024The Prompt Report: A Systematic Survey of Prompt Engineering Techniquescited by 121
  9. 2024Mitigating Catastrophic Forgetting in Large Language Models with Self-Synthesized Rehearsalcited by 118
  10. 2024Parrot: Efficient Serving of LLM-based Applications with Semantic Variablecited by 99
  11. 2022Learning to Break the Loop: Analyzing and Mitigating Repetitions for Neural Text Generationcited by 94
  12. 2021On Transferability of Prompt Tuning for Natural Language Processingcited by 75
  13. 2024Instructional Fingerprinting of Large Language Modelscited by 68
  14. 2023Prompt Stealing Attacks Against Text-to-Image Generation Modelscited by 51
  15. 2024PARDEN, Can You Repeat That? Defending against Jailbreaks via Repetitioncited by 35
  16. 2022Why So Toxic? Measuring and Triggering Toxic Behavior in Open-Domain Chatbotscited by 33
  17. 2023PromptCARE: Prompt Copyright Protection by Watermark Injection and Verificationcited by 32
  18. 2024Teach LLMs to Phish: Stealing Private Information from Language Modelscited by 32
  19. 2023AutoHint: Automatic Prompt Optimization with Hint Generationcited by 30
  20. 2024PEFTGuard: Detecting Backdoor Attacks Against Parameter-Efficient Fine-Tuningcited by 26
  21. 2019FAQ Retrieval using Query-Question Similarity and BERT-Based Query-Answer Relevancecited by 25
  22. 2024Citation-Enhanced Generation for LLM-based Chatbotscited by 25
  23. 2024Is Difficulty Calibration All We Need? Towards More Practical Membership Inference Attackscited by 21
  24. 2024PromSec: Prompt Optimization for Secure Generation of Functional Source Code with Large Language Models (LLMs)cited by 19
  25. 2025Probe before You Talk: Towards Black-box Defense against Backdoor Unalignment for Large Language Modelscited by 19
  26. 2025Rethinking Data Protection in the (Generative) Artificial Intelligence Eracited by 18
  27. 2024On the (In)Security of LLM App Storescited by 14
  28. 2025On Benchmarking Code LLMs for Android Malware Analysiscited by 13
  29. 2023Stateful Defenses for Machine Learning Models Are Not Yet Secure Against Black-box Attackscited by 11
  30. 2024PRSA: Prompt Stealing Attacks against Real-World Prompt Servicescited by 10
  31. 2024CoSearchAgent: A Lightweight Collaborative Search Agent with Large Language Modelscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2024Watermarking Language Models through Language Models
  2. 2025SWAP: Towards Copyright Auditing of Soft Prompts via Sequential Watermarking
  3. 2025SoK: Are Watermarks in LLMs Ready for Deployment?
  4. 2026PromptMark: A Prompt-Guided Iterative-Feedback Framework for Source Code Watermarking ↗
  5. 2024Can Watermarked LLMs be Identified by Users via Crafted Prompts?
  6. 2025Copyright Protection for Large Language Models: A Survey of Methods, Challenges, and Trends
  7. 2024Can Watermarking Large Language Models Prevent Copyrighted Text Generation and Hide Training Data?
  8. 2024Lost in Overlap: Exploring Logit-based Watermark Collision in LLMs