# In-Context Watermarks for Large Language Models

arXiv: https://arxiv.org/abs/2505.16934 · HTML: https://mlsanity.com/paper/2505.16934/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Watermarking generated content and models, one of 12 directions in LLM safety and security.
- Builds on 62 of its 89 references that are in the map.
- First posted on arXiv 22 May 2025.

## Placed in

- LLM safety and security / [Watermarking generated content and models](https://mlsanity.com/safety/watermarking-generated-content-and-models/)

## Builds on (most cited first, 30 of 62)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://mlsanity.com/paper/2302.12173/) (2023)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [Instruction-Following Evaluation for Large Language Models](https://mlsanity.com/paper/2311.07911/) (2023)
- [DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature](https://mlsanity.com/paper/2301.11305/) (2023)
- [A Watermark for Large Language Models](https://mlsanity.com/paper/2301.10226/) (2023)
- [Prompt Injection attack against LLM-integrated Applications](https://mlsanity.com/paper/2306.05499/) (2023)
- [ELI5: Long Form Question Answering](https://mlsanity.com/paper/1907.09190/) (2019)
- [Can AI-Generated Text be Reliably Detected?](https://mlsanity.com/paper/2303.11156/) (2023)
- [Formalizing and Benchmarking Prompt Injection Attacks and Defenses](https://mlsanity.com/paper/2310.12815/) (2023)
- [Robust Distortion-free Watermarks for Language Models](https://mlsanity.com/paper/2307.15593/) (2023)
- [Provable Robust Watermarking for AI-Generated Text](https://mlsanity.com/paper/2306.17439/) (2023)
- [The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning](https://mlsanity.com/paper/2312.01552/) (2023)
- [Undetectable Watermarks for Language Models](https://mlsanity.com/paper/2306.09194/) (2023)
- [Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews](https://mlsanity.com/paper/2403.07183/) (2024)
- [CycleResearcher: Improving Automated Research via Automated Review](https://mlsanity.com/paper/2411.00816/) (2024)
- [Who Wrote this Code? Watermarking for Code Generation](https://mlsanity.com/paper/2305.15060/) (2023)
- [Mapping the Increasing Use of LLMs in Scientific Papers](https://mlsanity.com/paper/2404.01268/) (2024)
- [Automatic and Universal Prompt Injection Attacks against Large Language Models](https://mlsanity.com/paper/2403.04957/) (2024)
- [Unbiased Watermark for Large Language Models](https://mlsanity.com/paper/2310.10669/) (2023)
- [A Survey of Text Watermarking in the Era of Large Language Models](https://mlsanity.com/paper/2312.07913/) (2023)
- [A Semantic Invariant Robust Watermark for Large Language Models](https://mlsanity.com/paper/2310.06356/) (2023)
- [MarkLLM: An Open-Source Toolkit for LLM Watermarking](https://mlsanity.com/paper/2405.10051/) (2024)
- [Protecting Language Generation Models via Invisible Watermarking](https://mlsanity.com/paper/2302.03162/) (2023)
- [On the Possibilities of AI-Generated Text Detection](https://mlsanity.com/paper/2304.04736/) (2023)
- [Three Bricks to Consolidate Watermarks for Large Language Models](https://mlsanity.com/paper/2308.00113/) (2023)

## Built on by (most cited first, 10 of 10)

- [Position: LLM Watermarking Should Align Stakeholders' Incentives for Practical Adoption](https://mlsanity.com/paper/2510.18333/) (2025)
- [DAMASHA: Detecting AI in Mixed Adversarial Texts via Segmentation with Human-interpretable Attribution](https://mlsanity.com/paper/2512.04838/) (2025)
- [Hidden Tail: Adversarial Image Causing Stealthy Resource Consumption in Vision-Language Models](https://mlsanity.com/paper/2508.18805/) (2025)
- [Can You Detect the Difference?](https://mlsanity.com/paper/2507.10475/) (2025)
- [TokenPure: Watermark Removal through Tokenized Appearance and Structural Guidance](https://mlsanity.com/paper/2512.01314/) (2025)
- [Watermark Forensics for Generative Models: An Information-Theoretic Perspective](https://mlsanity.com/paper/2607.13003/) (2026)
- [Functional Subspace Watermarking for Large Language Models](https://mlsanity.com/paper/2603.18793/) (2026)
- [Learning to Follow In-Context Watermark Instructions via Self-Distillation](https://mlsanity.com/paper/2608.29030/) (2026)
- [SemTrace: Source-Grounded Semantic Signatures for Tracing LLM Exposure to Protected Documents](https://mlsanity.com/paper/2608.29575/) (2026)
- [SkillMarker: A KGW-Based Watermarking Method for Agent Skill Ownership Verification](https://mlsanity.com/paper/2610.15725/) (2026)

## Abstract (arXiv)

The growing use of large language models (LLMs) for sensitive applications has highlighted the need for effective watermarking techniques to ensure the provenance and accountability of AI-generated text. However, most existing watermarking methods require access to the decoding process, limiting their applicability in real-world settings. One illustrative example is the use of LLMs by dishonest reviewers in the context of academic peer review, where conference organizers have no access to the model used but still need to detect AI-generated reviews. Motivated by this gap, we introduce In-Context Watermarking (ICW), which embeds watermarks into generated text solely through prompt engineering, leveraging LLMs' in-context learning and instruction-following abilities. We investigate four ICW strategies at different levels of granularity, each paired with a tailored detection method. We further examine the Indirect Prompt Injection (IPI) setting as a specific case study, in which watermarking is covertly triggered by modifying input documents such as academic manuscripts. Our experiments validate the feasibility of ICW as a model-agnostic, practical watermarking approach. Moreover, our findings suggest that as LLMs become more capable, ICW offers a promising direction for scalable and accessible content attribution. Our code is available at https://github.com/yepengliu/In-Context-Watermarks.