# Can Watermarking Large Language Models Prevent Copyrighted Text Generation and Hide Training Data?

arXiv: https://arxiv.org/abs/2407.17417 · HTML: https://mlsanity.com/paper/2407.17417/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Placed in Watermarking generated content and models, one of 12 directions in LLM safety and security.
- Builds on 38 of its 42 references that are in the map.
- First posted on arXiv 24 Jul 2024.

## Placed in

- LLM safety and security / [Watermarking generated content and models](https://mlsanity.com/safety/watermarking-generated-content-and-models/)

## Builds on (most cited first, 30 of 38)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Membership Inference Attacks From First Principles](https://mlsanity.com/paper/2112.03570/) (2021)
- [The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks](https://mlsanity.com/paper/1802.08232/) (2018)
- [Quantifying Memorization Across Neural Language Models](https://mlsanity.com/paper/2202.07646/) (2022)
- [Extracting Training Data from Diffusion Models](https://mlsanity.com/paper/2301.13188/) (2023)
- [A Watermark for Large Language Models](https://mlsanity.com/paper/2301.10226/) (2023)
- [Deduplicating Training Data Makes Language Models Better](https://mlsanity.com/paper/2107.06499/) (2021)
- [GPT-NeoX-20B: An Open-Source Autoregressive Language Model](https://mlsanity.com/paper/2204.06745/) (2022)
- [Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense](https://mlsanity.com/paper/2303.13408/) (2023)
- [Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models](https://mlsanity.com/paper/2212.03860/) (2022)
- [Detecting Pretraining Data from Large Language Models](https://mlsanity.com/paper/2310.16789/) (2023)
- [Robust Distortion-free Watermarks for Language Models](https://mlsanity.com/paper/2307.15593/) (2023)
- [Deduplicating Training Data Mitigates Privacy Risks in Language Models](https://mlsanity.com/paper/2202.06539/) (2022)
- [Provable Robust Watermarking for AI-Generated Text](https://mlsanity.com/paper/2306.17439/) (2023)
- [White-box vs Black-box: Bayes Optimal Strategies for Membership Inference](https://mlsanity.com/paper/1908.11229/) (2019)
- [Membership Inference Attacks against Language Models via Neighbourhood Comparison](https://mlsanity.com/paper/2305.18462/) (2023)
- [Proving Test Set Contamination in Black Box Language Models](https://mlsanity.com/paper/2310.17623/) (2023)
- [Undetectable Watermarks for Language Models](https://mlsanity.com/paper/2306.09194/) (2023)
- [Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models](https://mlsanity.com/paper/2205.10770/) (2022)
- [Understanding and Mitigating Copying in Diffusion Models](https://mlsanity.com/paper/2305.20086/) (2023)
- [Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks](https://mlsanity.com/paper/2203.03929/) (2022)
- [A Semantic Invariant Robust Watermark for Large Language Models](https://mlsanity.com/paper/2310.06356/) (2023)
- [Copyright Violations and Large Language Models](https://mlsanity.com/paper/2310.13771/) (2023)
- [Min-K%++: Improved Baseline for Detecting Pre-Training Data from Large Language Models](https://mlsanity.com/paper/2404.02936/) (2024)
- [Preventing Verbatim Memorization in Language Models Gives a False Sense of Privacy](https://mlsanity.com/paper/2210.17546/) (2022)
- [On Provable Copyright Protection for Generative Models](https://mlsanity.com/paper/2302.10870/) (2023)

## Built on by (most cited first, 18 of 18)

- [Dataset Protection via Watermarked Canaries in Retrieval-Augmented LLMs](https://mlsanity.com/paper/2502.10673/) (2025)
- [Marking Code Without Breaking It: Code Watermarking for Detecting LLM-Generated Code](https://mlsanity.com/paper/2502.18851/) (2025)
- [In-Context Watermarks for Large Language Models](https://mlsanity.com/paper/2505.16934/) (2025)
- [Position: LLM Watermarking Should Align Stakeholders' Incentives for Practical Adoption](https://mlsanity.com/paper/2510.18333/) (2025)
- [WaterMod: Modular Token-Rank Partitioning for Probability-Balanced LLM Watermarking](https://mlsanity.com/paper/2511.07863/) (2025)
- [Nob-MIAs: Non-biased Membership Inference Attacks Assessment on Large Language Models with Ex-Post Dataset Construction](https://mlsanity.com/paper/2408.05968/) (2024)
- [Large Language Model Sourcing: A Survey](https://mlsanity.com/paper/2510.10161/) (2025)
- [Beyond Easy Wins: A Text Hardness-Aware Benchmark for LLM-generated Text Detection](https://mlsanity.com/paper/2507.15286/) (2025)
- [ExpShield: Safeguarding Web Text from Unauthorized Crawling and LLM Exploitation](https://mlsanity.com/paper/2412.21123/) (2024)
- [Imitative Membership Inference Attack](https://mlsanity.com/paper/2509.06796/) (2025)
- [Inner-Probe: Discovering Copyright-related Data Generation in LLM Architecture](https://mlsanity.com/paper/2410.04454/) (2024)
- [HeavyWater and SimplexWater: Distortion-Free LLM Watermarks for Low-Entropy Next-Token Predictions](https://mlsanity.com/paper/2506.06409/) (2025)
- [AI Security in the Foundation Model Era: A Comprehensive Survey from a Unified Perspective](https://mlsanity.com/paper/2603.24857/) (2026)
- [Pretraining Data Exposure in Large Language Models: A Survey of Membership Inference, Data Contamination, and Security Implications](https://mlsanity.com/paper/2605.26133/) (2026)
- [Perturb Your Data: Paraphrase-Guided Training Data Watermarking](https://mlsanity.com/paper/2512.17075/) (2025)
- [DependencyAI: Detecting AI Generated Text through Dependency Parsing](https://mlsanity.com/paper/2602.15514/) (2026)
- [Efficient and Scalable Provenance Tracking for LLM-Generated Code Snippets](https://mlsanity.com/paper/2605.28510/) (2026)
- [Can Watermarking Techniques Help Prevent LLM Model Stealing?](https://mlsanity.com/paper/2607.10794/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have demonstrated impressive capabilities in generating diverse and contextually rich text. However, concerns regarding copyright infringement arise as LLMs may inadvertently produce copyrighted material. In this paper, we first investigate the effectiveness of watermarking LLMs as a deterrent against the generation of copyrighted texts. Through theoretical analysis and empirical evaluation, we demonstrate that incorporating watermarks into LLMs significantly reduces the likelihood of generating copyrighted content, thereby addressing a critical concern in the deployment of LLMs. However, we also find that watermarking can have unintended consequences on Membership Inference Attacks (MIAs), which aim to discern whether a sample was part of the pretraining dataset and may be used to detect copyright violations. Surprisingly, we find that watermarking adversely affects the success rate of MIAs, complicating the task of detecting copyrighted text in the pretraining dataset. These results reveal the complex interplay between different regulatory measures, which may impact each other in unforeseen ways. Finally, we propose an adaptive technique to improve the success rate of a recent MIA under watermarking. Our findings underscore the importance of developing adaptive methods to study critical problems in LLMs with potential legal implications.