# Proving membership in LLM pretraining data via data watermarks

arXiv: https://arxiv.org/abs/2402.10892 · HTML: https://mlsanity.com/paper/2402.10892/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Placed in Watermarking generated content and models, one of 12 directions in LLM safety and security.
- Builds on 22 of its 35 references that are in the map.
- First posted on arXiv 16 Feb 2024.

## Placed in

- LLM safety and security / [Watermarking generated content and models](https://mlsanity.com/safety/watermarking-generated-content-and-models/)

## Builds on (most cited first, 22 of 22)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://mlsanity.com/paper/2302.12173/) (2023)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [Membership Inference Attacks From First Principles](https://mlsanity.com/paper/2112.03570/) (2021)
- [The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks](https://mlsanity.com/paper/1802.08232/) (2018)
- [Deduplicating Training Data Mitigates Privacy Risks in Language Models](https://mlsanity.com/paper/2202.06539/) (2022)
- [Data Contamination: From Memorization to Exploitation](https://mlsanity.com/paper/2203.08242/) (2022)
- [Proving Test Set Contamination in Black Box Language Models](https://mlsanity.com/paper/2310.17623/) (2023)
- [BadNL: Backdoor Attacks against NLP Models with Semantic-preserving Improvements](https://mlsanity.com/paper/2006.01043/) (2020)
- [Counterfactual Memorization in Neural Language Models](https://mlsanity.com/paper/2112.12938/) (2021)
- [The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset](https://mlsanity.com/paper/2303.03915/) (2023)
- [Radioactive data: tracing through training](https://mlsanity.com/paper/2002.00937/) (2020)
- [Mitigating Dataset Harms Requires Stewardship: Lessons from 1000 Papers](https://mlsanity.com/paper/2108.02922/) (2021)
- [Did You Train on My Dataset? Towards Public Dataset Protection with Clean-Label Backdoor Watermarking](https://mlsanity.com/paper/2303.11470/) (2023)
- [Data Portraits: Recording Foundation Model Training Data](https://mlsanity.com/paper/2303.03919/) (2023)
- [Reclaiming the Digital Commons: A Public Data Trust for Training Data](https://mlsanity.com/paper/2303.09001/) (2023)

## Built on by (most cited first, 30 of 40)

- [SoK: Membership Inference Attacks on LLMs are Rushing Nowhere (and How to Fix It)](https://mlsanity.com/paper/2406.17975/) (2024)
- [SoK: Watermarking for AI-Generated Content](https://mlsanity.com/paper/2411.18479/) (2024)
- [Evaluating Copyright Takedown Methods for Language Models](https://mlsanity.com/paper/2406.18664/) (2024)
- [Does Data Contamination Detection Work (Well) for LLMs? A Survey and Evaluation on Detection Assumptions](https://mlsanity.com/paper/2410.18966/) (2024)
- [Watermarking Techniques for Large Language Models: A Survey](https://mlsanity.com/paper/2409.00089/) (2024)
- [The Canary's Echo: Auditing Privacy Risks of LLM-Generated Synthetic Text](https://mlsanity.com/paper/2502.14921/) (2025)
- [A General Framework for Data-Use Auditing of ML Models](https://mlsanity.com/paper/2407.15100/) (2024)
- [Ward: Provable RAG Dataset Inference via LLM Watermarks](https://mlsanity.com/paper/2410.03537/) (2024)
- [Robust Data Watermarking in Language Models by Injecting Fictitious Knowledge](https://mlsanity.com/paper/2503.04036/) (2025)
- [The Mosaic Memory of Large Language Models](https://mlsanity.com/paper/2405.15523/) (2024)
- [STAMP Your Content: Proving Dataset Membership via Watermarked Rephrasings](https://mlsanity.com/paper/2504.13416/) (2025)
- [DATABench: Evaluating Dataset Auditing in Deep Learning from an Adversarial Perspective](https://mlsanity.com/paper/2507.05622/) (2025)
- [Self-Comparison for Dataset-Level Membership Inference in Large (Vision-)Language Models](https://mlsanity.com/paper/2410.13088/) (2024)
- [Position: LLM Watermarking Should Align Stakeholders' Incentives for Practical Adoption](https://mlsanity.com/paper/2510.18333/) (2025)
- [Memorization in In-Context Learning](https://mlsanity.com/paper/2408.11546/) (2024)
- [Winter Soldier: Backdooring Language Models at Pre-Training with Indirect Data Poisoning](https://mlsanity.com/paper/2506.14913/) (2025)
- [Nob-MIAs: Non-biased Membership Inference Attacks Assessment on Large Language Models with Ex-Post Dataset Construction](https://mlsanity.com/paper/2408.05968/) (2024)
- [Leave No TRACE: Black-box Detection of Copyrighted Dataset Usage in Large Language Models via Watermarking](https://mlsanity.com/paper/2510.02962/) (2025)
- [TRACE: TRansformer-based Attribution using Contrastive Embeddings in LLMs](https://mlsanity.com/paper/2407.04981/) (2024)
- [Instance-Level Data-Use Auditing of Visual ML Models](https://mlsanity.com/paper/2503.22413/) (2025)
- [Data Provenance Auditing of Fine-Tuned Large Language Models with a Text-Preserving Technique](https://mlsanity.com/paper/2510.09655/) (2025)
- [ImageSentinel: Protecting Visual Datasets from Unauthorized Retrieval-Augmented Image Generation](https://mlsanity.com/paper/2510.12119/) (2025)
- [ContextLeak: Auditing Leakage in Private In-Context Learning Methods](https://mlsanity.com/paper/2512.16059/) (2025)
- [Watermarking LLM Agent Trajectories](https://mlsanity.com/paper/2602.18700/) (2026)
- [Protecting Copyrighted Material with Unique Identifiers in Large Language Model Training](https://mlsanity.com/paper/2403.15740/) (2024)
- [Large Language Models Memorize Sensor Datasets! Implications on Human Activity Recognition Research](https://mlsanity.com/paper/2406.05900/) (2024)
- [ExpShield: Safeguarding Web Text from Unauthorized Crawling and LLM Exploitation](https://mlsanity.com/paper/2412.21123/) (2024)
- [Large Language Models as Search Engines: Societal Challenges](https://mlsanity.com/paper/2512.08946/) (2025)
- [Watermarking Large Language Models and the Generated Content: Opportunities and Challenges](https://mlsanity.com/paper/2410.19096/) (2024)
- [C2LEVA: Toward Comprehensive and Contamination-Free Language Model Evaluation](https://mlsanity.com/paper/2412.04947/) (2024)

## Abstract (arXiv)

Detecting whether copyright holders' works were used in LLM pretraining is poised to be an important problem. This work proposes using data watermarks to enable principled detection with only black-box model access, provided that the rightholder contributed multiple training documents and watermarked them before public release. By applying a randomly sampled data watermark, detection can be framed as hypothesis testing, which provides guarantees on the false detection rate. We study two watermarks: one that inserts random sequences, and another that randomly substitutes characters with Unicode lookalikes. We first show how three aspects of watermark design -- watermark length, number of duplications, and interference -- affect the power of the hypothesis test. Next, we study how a watermark's detection strength changes under model and dataset scaling: while increasing the dataset size decreases the strength of the watermark, watermarks remain strong if the model size also increases. Finally, we view SHA hashes as natural watermarks and show that we can robustly detect hashes from BLOOM-176B's training data, as long as they occurred at least 90 times. Together, our results point towards a promising future for data watermarks in real world use.