# Perturb Your Data: Paraphrase-Guided Training Data Watermarking

arXiv: https://arxiv.org/abs/2512.17075 · HTML: https://mlsanity.com/paper/2512.17075/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 35 references that are in the map.
- First posted on arXiv 18 Dec 2025.

## Builds on (most cited first, 21 of 21)

- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research](https://mlsanity.com/paper/2402.00159/) (2024)
- [Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense](https://mlsanity.com/paper/2303.13408/) (2023)
- [Proving Test Set Contamination in Black Box Language Models](https://mlsanity.com/paper/2310.17623/) (2023)
- [LLM Dataset Inference: Did you train on my dataset?](https://mlsanity.com/paper/2406.06443/) (2024)
- [Blind Baselines Beat Membership Inference Attacks for Foundation Models](https://mlsanity.com/paper/2406.16201/) (2024)
- [Scalable Membership Inference Attacks via Quantile Regression](https://mlsanity.com/paper/2307.03694/) (2023)
- [Pretraining Data Detection for Large Language Models: A Divergence-based Calibration Method](https://mlsanity.com/paper/2409.14781/) (2024)
- [Paloma: A Benchmark for Evaluating Language Model Fit](https://mlsanity.com/paper/2312.10523/) (2023)
- [BITE: Textual Backdoor Attacks with Iterative Trigger Injection](https://mlsanity.com/paper/2205.12700/) (2022)
- [Proving membership in LLM pretraining data via data watermarks](https://mlsanity.com/paper/2402.10892/) (2024)
- [The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text](https://mlsanity.com/paper/2506.05209/) (2025)
- [Can Watermarking Large Language Models Prevent Copyrighted Text Generation and Hide Training Data?](https://mlsanity.com/paper/2407.17417/) (2024)
- [Fine-Tuning Language Models with Differential Privacy through Adaptive Noise Allocation](https://mlsanity.com/paper/2410.02912/) (2024)
- [Towards a consistent interpretation of AIOps models](https://mlsanity.com/paper/2202.02298/) (2022)
- [STAMP Your Content: Proving Dataset Membership via Watermarked Rephrasings](https://mlsanity.com/paper/2504.13416/) (2025)
- [Unlocking Post-hoc Dataset Inference with Synthetic Data](https://mlsanity.com/paper/2506.15271/) (2025)
- [Winter Soldier: Backdooring Language Models at Pre-Training with Indirect Data Poisoning](https://mlsanity.com/paper/2506.14913/) (2025)
- [Beyond Public Access in LLM Pre-Training Data](https://mlsanity.com/paper/2505.00020/) (2025)

## Abstract (arXiv)

Training data detection is critical for enforcing copyright and data licensing, as Large Language Models (LLM) are trained on massive text corpora scraped from the internet. We present SPECTRA, a watermarking approach that makes training data reliably detectable even when it comprises less than 0.001% of the training corpus. SPECTRA works by paraphrasing text using an LLM and assigning a score based on how likely each paraphrase is, according to a separate scoring model. A paraphrase is chosen so that its score closely matches that of the original text, to avoid introducing any distribution shifts. To test whether a suspect model has been trained on the watermarked data, we compare its token probabilities against those of the scoring model. We demonstrate that SPECTRA achieves a consistent p-value gap of over nine orders of magnitude when detecting data used for training versus data not used for training, which is greater than all baselines tested. SPECTRA equips data owners with a scalable, deploy-before-release watermark that survives even large-scale LLM training.