mlsanityData to 7 Oct 2026

Proving membership in LLM pretraining data via data watermarks

arXiv 2402.10892 ↗First posted 16 Feb 2024cs.CR cs.CL cs.LGcited by 38

On the map: LLM safety and security → Watermarking generated content and models

Abstract (arXiv)

Detecting whether copyright holders' works were used in LLM pretraining is poised to be an important problem. This work proposes using data watermarks to enable principled detection with only black-box model access, provided that the rightholder contributed multiple training documents and watermarked them before public release. By applying a randomly sampled data watermark, detection can be framed as hypothesis testing, which provides guarantees on the false detection rate. We study two watermarks: one that inserts random sequences, and another that randomly substitutes characters with Unicode lookalikes. We first show how three aspects of watermark design -- watermark length, number of duplications, and interference -- affect the power of the hypothesis test. Next, we study how a watermark's detection strength changes under model and dataset scaling: while increasing the dataset size decreases the strength of the watermark, watermarks remain strong if the model size also increases. Finally, we view SHA hashes as natural watermarks and show that we can robustly detect hashes from BLOOM-176B's training data, as long as they occurred at least 90 times. Together, our results point towards a promising future for data watermarks in real world use.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Placed in Watermarking generated content and models, one of 12 directions in LLM safety and security. Builds on 22 of its 35 references that are in the map. First posted on arXiv 16 Feb 2024.

Builds on

22 of 35 references.

References · 22

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2016Deep Learning with Differential Privacycited by 2,682
  4. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  5. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  6. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  7. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  8. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  9. 2018Model Cards for Model Reportingcited by 1,068
  10. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  11. 2021Membership Inference Attacks From First Principlescited by 839
  12. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 309
10 more
  1. 2022Data Contamination: From Memorization to Exploitationcited by 234
  2. 2023Proving Test Set Contamination in Black Box Language Modelscited by 227
  3. 2020BadNL: Backdoor Attacks against NLP Models with Semantic-preserving Improvementscited by 191
  4. 2021Counterfactual Memorization in Neural Language Modelscited by 155
  5. 2023The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Datasetcited by 148
  6. 2020Radioactive data: tracing through trainingcited by 67
  7. 2021Mitigating Dataset Harms Requires Stewardship: Lessons from 1000 Paperscited by 49
  8. 2023Did You Train on My Dataset? Towards Public Dataset Protection with Clean-Label Backdoor Watermarkingcited by 37
  9. 2023Data Portraits: Recording Foundation Model Training Datacited by 29
  10. 2023Reclaiming the Digital Commons: A Public Data Trust for Training Datacited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 18; 2026: 5.

38 papers.

Sort:

By area:

  1. 2024SoK: Membership Inference Attacks on LLMs are Rushing Nowhere (and How to Fix It)cited by 54
  2. 2024SoK: Watermarking for AI-Generated Contentcited by 43
  3. 2024Evaluating Copyright Takedown Methods for Language Modelscited by 42
  4. 2024Does Data Contamination Detection Work (Well) for LLMs? A Survey and Evaluation on Detection Assumptionscited by 26
  5. 2024Watermarking Techniques for Large Language Models: A Surveycited by 20
  6. 2025The Canary's Echo: Auditing Privacy Risks of LLM-Generated Synthetic Textcited by 18
  7. 2024A General Framework for Data-Use Auditing of ML Modelscited by 15
  8. 2024Ward: Provable RAG Dataset Inference via LLM Watermarkscited by 15
  9. 2024The Mosaic Memory of Large Language Modelscited by 9
  10. 2025Robust Data Watermarking in Language Models by Injecting Fictitious Knowledgecited by 8
  11. 2025STAMP Your Content: Proving Dataset Membership via Watermarked Rephrasingscited by 8
  12. 2025DATABench: Evaluating Dataset Auditing in Deep Learning from an Adversarial Perspectivecited by 8
26 more
  1. 2025Position: LLM Watermarking Should Align Stakeholders' Incentives for Practical Adoptioncited by 7
  2. 2024Memorization in In-Context Learningcited by 6
  3. 2024Self-Comparison for Dataset-Level Membership Inference in Large (Vision-)Language Modelscited by 6
  4. 2025Winter Soldier: Backdooring Language Models at Pre-Training with Indirect Data Poisoningcited by 6
  5. 2025Leave No TRACE: Black-box Detection of Copyrighted Dataset Usage in Large Language Models via Watermarkingcited by 5
  6. 2024TRACE: TRansformer-based Attribution using Contrastive Embeddings in LLMscited by 4
  7. 2025Instance-Level Data-Use Auditing of Visual ML Modelscited by 4
  8. 2025ImageSentinel: Protecting Visual Datasets from Unauthorized Retrieval-Augmented Image Generationcited by 4
  9. 2025ContextLeak: Auditing Leakage in Private In-Context Learning Methodscited by 4
  10. 2024Large Language Models Memorize Sensor Datasets! Implications on Human Activity Recognition Researchcited by 3
  11. 2025Data Provenance Auditing of Fine-Tuned Large Language Models with a Text-Preserving Techniquecited by 3
  12. 2025Large Language Models as Search Engines: Societal Challengescited by 3
  13. 2026Watermarking LLM Agent Trajectoriescited by 3
  14. 2024Protecting Copyrighted Material with Unique Identifiers in Large Language Model Trainingcited by 2
  15. 2024Watermarking Large Language Models and the Generated Content: Opportunities and Challengescited by 2
  16. 2024C2LEVA: Toward Comprehensive and Contamination-Free Language Model Evaluationcited by 2
  17. 2025Interrogating LLM design under a fair learning doctrinecited by 1
  18. 2025LexiMark: Robust Watermarking via Lexical Substitutions to Enhance Membership Verification of an LLM's Textual Training Datacited by 1
  19. 2025TokenSmith: Streamlining Data Editing, Search, and Inspection for Large-Scale Language Model Training and Interpretabilitycited by 0
  20. 2025EL-MIA: Quantifying Membership Inference Risks of Sensitive Entities in LLMscited by 0
  21. 2025Watermarking Large Language Models in Europe: Interpreting the AI Act in Light of Technologycited by 0
  22. 2025Perturb Your Data: Paraphrase-Guided Training Data Watermarkingcited by 0
  23. 2026The Limits of AI Data Transparency Policy: Three Disclosure Fallaciescited by 0
  24. 2026Dataset Watermarking with Provable Black-Box Detectioncited by 0
  25. 2026Correcting test set contamination by spiking the training datacited by 0
  26. 2026Phantoms and Disclosures: A Statistical Framework for Auditing Privacy in Synthetic Datacited by 0