# Dataset Watermarking with Provable Black-Box Detection

arXiv: https://arxiv.org/abs/2605.06865 · HTML: https://mlsanity.com/paper/2605.06865/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 37 references that are in the map.
- First posted on arXiv 7 May 2026.

## Builds on (most cited first, 26 of 26)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence](https://mlsanity.com/paper/2606.19348/) (2026)
- [A Watermark for Large Language Models](https://mlsanity.com/paper/2301.10226/) (2023)
- [Detecting Pretraining Data from Large Language Models](https://mlsanity.com/paper/2310.16789/) (2023)
- [Noise or Signal: The Role of Image Backgrounds in Object Recognition](https://mlsanity.com/paper/2006.09994/) (2020)
- [Proving Test Set Contamination in Black Box Language Models](https://mlsanity.com/paper/2310.17623/) (2023)
- [A Careful Examination of Large Language Model Performance on Grade School Arithmetic](https://mlsanity.com/paper/2405.00332/) (2024)
- [On the Reliability of Watermarks for Large Language Models](https://mlsanity.com/paper/2306.04634/) (2023)
- [Investigating Data Contamination for Pre-training Language Models](https://mlsanity.com/paper/2401.06059/) (2024)
- [Watermarks in the Sand: Impossibility of Strong Watermarking for Generative Models](https://mlsanity.com/paper/2311.04378/) (2023)
- [Proving membership in LLM pretraining data via data watermarks](https://mlsanity.com/paper/2402.10892/) (2024)
- [Understanding Rare Spurious Correlations in Neural Networks](https://mlsanity.com/paper/2202.05189/) (2022)
- [Do SSL Models Have D'ej`a Vu? A Case of Unintended Memorization in Self-supervised Learning](https://mlsanity.com/paper/2304.13850/) (2023)
- [Waterfall: Framework for Robust and Scalable Text Watermarking and Provenance for LLMs](https://mlsanity.com/paper/2407.04411/) (2024)
- [Robust Data Watermarking in Language Models by Injecting Fictitious Knowledge](https://mlsanity.com/paper/2503.04036/) (2025)
- [STAMP Your Content: Proving Dataset Membership via Watermarked Rephrasings](https://mlsanity.com/paper/2504.13416/) (2025)
- [Detecting Benchmark Contamination Through Watermarking](https://mlsanity.com/paper/2502.17259/) (2025)
- [Leave No TRACE: Black-box Detection of Copyrighted Dataset Usage in Large Language Models via Watermarking](https://mlsanity.com/paper/2510.02962/) (2025)
- [SLIM: Stealthy Low-Coverage Black-Box Watermarking via Latent-Space Confusion Zones](https://mlsanity.com/paper/2601.03242/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are often pre-trained and post-trained on vast amounts of loosely curated data, and potentially have been trained on proprietary datasets or the benchmarks used for evaluation. This motivates dataset watermarking: designing datasets such that training on them leaves detectable signatures in the resulting model. Existing approaches embed watermarks by either injecting artificial content or paraphrasing full examples under token-level generation control, and many require model access beyond generated outputs for detection. Our method embeds a dataset-level watermark by increasing the co-occurrence of randomly selected word pairs through meaning-preserving local edits, and detects it from generated text alone with provable false positive control. We conduct evaluation on four base models and three datasets and show that it reliably detects the watermark (p <0.01) in the fine-tuning stage, outperforming existing black-box baselines. Our method remains effective under substantial dilution where watermarked data is less than 5% of fine-tuning tokens. Moreover, our method better preserves benchmark utility and semantic integrity than existing baselines.