# Mark Your LLM: Detecting the Misuse of Open-Source Large Language Models via Watermarking

arXiv: https://arxiv.org/abs/2503.04636 · HTML: https://mlsanity.com/paper/2503.04636/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Placed in Watermarking generated content and models, one of 12 directions in LLM safety and security.
- Builds on 48 of its 60 references that are in the map.
- First posted on arXiv 6 Mar 2025.

## Placed in

- LLM safety and security / [Watermarking generated content and models](https://mlsanity.com/safety/watermarking-generated-content-and-models/)

## Builds on (most cited first, 30 of 48)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [A Watermark for Large Language Models](https://mlsanity.com/paper/2301.10226/) (2023)
- [Neural Text Generation with Unlikelihood Training](https://mlsanity.com/paper/1908.04319/) (2019)
- [Can AI-Generated Text be Reliably Detected?](https://mlsanity.com/paper/2303.11156/) (2023)
- [Robust Distortion-free Watermarks for Language Models](https://mlsanity.com/paper/2307.15593/) (2023)
- [Evaluating Large Language Models at Evaluating Instruction Following](https://mlsanity.com/paper/2310.07641/) (2023)
- [Provable Robust Watermarking for AI-Generated Text](https://mlsanity.com/paper/2306.17439/) (2023)
- [Reasoning with Language Model Prompting: A Survey](https://mlsanity.com/paper/2212.09597/) (2022)
- [Who Wrote this Code? Watermarking for Code Generation](https://mlsanity.com/paper/2305.15060/) (2023)
- [Pretrained Language Models for Text Generation: A Survey](https://mlsanity.com/paper/2201.05273/) (2022)
- [A Semantic Invariant Robust Watermark for Large Language Models](https://mlsanity.com/paper/2310.06356/) (2023)
- [A Survey of Text Watermarking in the Era of Large Language Models](https://mlsanity.com/paper/2312.07913/) (2023)
- [Unbiased Watermark for Large Language Models](https://mlsanity.com/paper/2310.10669/) (2023)
- [Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for Large Language Models](https://mlsanity.com/paper/2305.14710/) (2023)
- [Combating Misinformation in the Age of LLMs: Opportunities and Challenges](https://mlsanity.com/paper/2311.05656/) (2023)
- [MarkLLM: An Open-Source Toolkit for LLM Watermarking](https://mlsanity.com/paper/2405.10051/) (2024)
- [Watermark Stealing in Large Language Models](https://mlsanity.com/paper/2402.19361/) (2024)
- [An Entropy-based Text Watermarking Detection Method](https://mlsanity.com/paper/2403.13485/) (2024)
- [Instructional Fingerprinting of Large Language Models](https://mlsanity.com/paper/2401.12255/) (2024)
- [An Unforgeable Publicly Verifiable Watermark for Large Language Models](https://mlsanity.com/paper/2307.16230/) (2023)

## Built on by (most cited first, 19 of 19)

- [GVPO: Group Variance Policy Optimization for Large Language Model Post-Training](https://mlsanity.com/paper/2504.19599/) (2025)
- [Copyright Protection for Large Language Models: A Survey of Methods, Challenges, and Trends](https://mlsanity.com/paper/2508.11548/) (2025)
- [SoK: Large Language Model Copyright Auditing via Fingerprinting](https://mlsanity.com/paper/2508.19843/) (2025)
- [Watermarking Diffusion Language Models](https://mlsanity.com/paper/2509.24368/) (2025)
- [LLM Fingerprinting via Semantically Conditioned Watermarks](https://mlsanity.com/paper/2505.16723/) (2025)
- [Position: LLM Watermarking Should Align Stakeholders' Incentives for Practical Adoption](https://mlsanity.com/paper/2510.18333/) (2025)
- [A Reinforcement Learning Framework for Robust and Secure LLM Watermarking](https://mlsanity.com/paper/2510.21053/) (2025)
- [iSeal: Encrypted Fingerprinting for Reliable LLM Ownership Verification](https://mlsanity.com/paper/2511.08905/) (2025)
- [PathMark: Protecting Intellectual Property of Mixture-of-Expert LLMs via Path Watermarks](https://mlsanity.com/paper/2607.03688/) (2026)
- [Your "Pro" LLM Subscription May Actually Be "Free": Exposing Fingerprint Spoofing Risks in LLM Inference Services](https://mlsanity.com/paper/2606.16100/) (2026)
- [Towards Provable (In)Secure Model Weight Release Schemes](https://mlsanity.com/paper/2506.19874/) (2025)
- [ForgetMark: Stealthy Fingerprint Embedding via Targeted Unlearning in Language Models](https://mlsanity.com/paper/2601.08189/) (2026)
- [Implicit Identity Technologies for LLMs: Fingerprinting and Watermarking across Datasets, Models, and Generated Content](https://mlsanity.com/paper/2605.29245/) (2026)
- [Targeted Counterfactual Fingerprinting for Black-Box LLM Ownership Verification](https://mlsanity.com/paper/2608.08195/) (2026)
- [Large Language Models and Forensic Linguistics: Navigating Opportunities and Threats in the Age of Generative AI](https://mlsanity.com/paper/2512.06922/) (2025)
- [Against Political Polarization: A Unified Framework for Tracing Evolving Political Ideologies on Social Media](https://mlsanity.com/paper/2608.17987/) (2026)
- [The Tokens Remember: When Tokenization Bypasses Knowledge Editing and Unlearning](https://mlsanity.com/paper/2609.29045/) (2026)
- [The Fragility of Trigger-Tag Mechanisms for Misuse Detection in Open-Weight LLMs](https://mlsanity.com/paper/2610.03124/) (2026)
- [Radioactive Feedback: Tracing Models Trained by Label-Only LLM Supervision](https://mlsanity.com/paper/2610.15685/) (2026)

## Abstract (arXiv)

As open-source large language models (LLMs) like Llama3 become more capable, it is crucial to develop watermarking techniques to detect their potential misuse. Existing watermarking methods either add watermarks during LLM inference, which is unsuitable for open-source LLMs, or primarily target classification LLMs rather than recent generative LLMs. Adapting these watermarks to open-source LLMs for misuse detection remains an open challenge. This work defines two misuse scenarios for open-source LLMs: intellectual property (IP) violation and LLM Usage Violation. Then, we explore the application of inference-time watermark distillation and backdoor watermarking in these contexts. We propose comprehensive evaluation methods to assess the impact of various real-world further fine-tuning scenarios on watermarks and the effect of these watermarks on LLM performance. Our experiments reveal that backdoor watermarking could effectively detect IP Violation, while inference-time watermark distillation is applicable in both scenarios but less robust to further fine-tuning and has a more significant impact on LLM performance compared to backdoor watermarking. Exploring more advanced watermarking methods for open-source LLMs to detect their misuse should be an important future direction.