# Language Models May Verbatim Complete Text They Were Not Explicitly Trained On

arXiv: https://arxiv.org/abs/2503.17514 · HTML: https://mlsanity.com/paper/2503.17514/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 71 references that are in the map.
- First posted on arXiv 21 Mar 2025.

## Builds on (most cited first, 30 of 49)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [PaLM 2 Technical Report](https://mlsanity.com/paper/2305.10403/) (2023)
- [The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale](https://mlsanity.com/paper/2406.17557/) (2024)
- [Machine Unlearning](https://mlsanity.com/paper/1912.03817/) (2019)
- [Membership Inference Attacks From First Principles](https://mlsanity.com/paper/2112.03570/) (2021)
- [The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks](https://mlsanity.com/paper/1802.08232/) (2018)
- [Quantifying Memorization Across Neural Language Models](https://mlsanity.com/paper/2202.07646/) (2022)
- [Extracting Training Data from Diffusion Models](https://mlsanity.com/paper/2301.13188/) (2023)
- [Deduplicating Training Data Makes Language Models Better](https://mlsanity.com/paper/2107.06499/) (2021)
- [ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Models](https://mlsanity.com/paper/1806.01246/) (2018)
- [Scalable Extraction of Training Data from (Production) Language Models](https://mlsanity.com/paper/2311.17035/) (2023)
- [Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models](https://mlsanity.com/paper/2212.03860/) (2022)
- [Label-Only Membership Inference Attacks](https://mlsanity.com/paper/2007.14321/) (2020)
- [Detecting Pretraining Data from Large Language Models](https://mlsanity.com/paper/2310.16789/) (2023)
- [NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark](https://mlsanity.com/paper/2310.18018/) (2023)
- [Deduplicating Training Data Mitigates Privacy Risks in Language Models](https://mlsanity.com/paper/2202.06539/) (2022)
- [White-box vs Black-box: Bayes Optimal Strategies for Membership Inference](https://mlsanity.com/paper/1908.11229/) (2019)
- [Membership Inference Attacks against Language Models via Neighbourhood Comparison](https://mlsanity.com/paper/2305.18462/) (2023)
- [Rethinking Machine Unlearning for Large Language Models](https://mlsanity.com/paper/2402.08787/) (2024)
- [Proving Test Set Contamination in Black Box Language Models](https://mlsanity.com/paper/2310.17623/) (2023)
- [Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models](https://mlsanity.com/paper/2205.10770/) (2022)

## Built on by (most cited first, 21 of 21)

- [How much do language models memorize?](https://mlsanity.com/paper/2505.24832/) (2025)
- [FlexOlmo: Open Language Models for Flexible Data Use](https://mlsanity.com/paper/2507.07024/) (2025)
- [Train Once, Answer All: Many Pretraining Experiments for the Cost of One](https://mlsanity.com/paper/2509.23383/) (2025)
- [Antidistillation Fingerprinting](https://mlsanity.com/paper/2602.03812/) (2026)
- [Distributional Machine Unlearning via Selective Data Removal](https://mlsanity.com/paper/2507.15112/) (2025)
- [What Really is a Member? Discrediting Membership Inference via Poisoning](https://mlsanity.com/paper/2506.06003/) (2025)
- [A Unified Framework for Adversary-Aware Differential Privacy Bounds](https://mlsanity.com/paper/2507.08158/) (2025)
- [Quantifying the Effect of Test Set Contamination on Generative Evaluations](https://mlsanity.com/paper/2601.04301/) (2026)
- [A Prior-Aware Metric for Efficiently Distinguishing Memorization from Generalization in Large Language Models](https://mlsanity.com/paper/2602.18733/) (2026)
- [Beyond Indistinguishability: Measuring Extraction Risk in LLM APIs](https://mlsanity.com/paper/2604.18697/) (2026)
- [Statistical Unlearning of Distributions: A Hypothesis Testing Approach](https://mlsanity.com/paper/2605.16645/) (2026)
- [Nine Ways to Break Copyright Law and Why Our LLM Won't: A Fair Use Aligned Generation Framework](https://mlsanity.com/paper/2505.23788/) (2025)
- [FictionalQA: A Dataset for Studying Memorization and Knowledge Acquisition](https://mlsanity.com/paper/2506.05639/) (2025)
- [Counterfactual Influence as a Distributional Quantity](https://mlsanity.com/paper/2506.20481/) (2025)
- [The Limits of AI Data Transparency Policy: Three Disclosure Fallacies](https://mlsanity.com/paper/2601.18127/) (2026)
- [Learning to Detect Language Model Training Data via Active Reconstruction](https://mlsanity.com/paper/2602.19020/) (2026)
- [Watermarking for Proprietary Dataset Protection](https://mlsanity.com/paper/2607.00325/) (2026)
- [FindMyText: Robust, Scalable Detection of Text Containment in Large Web-Crawled Corpora](https://mlsanity.com/paper/2607.10020/) (2026)
- [Extractable Memorization From First Principles](https://mlsanity.com/paper/2607.12649/) (2026)
- [What to Forget in Unlearning? Forget Set Curation for Language Models](https://mlsanity.com/paper/2608.14855/) (2026)
- [Learning What to Forget: Distributional Unlearning for LLM Representation Spaces](https://mlsanity.com/paper/2609.38929/) (2026)

## Abstract (arXiv)

An important question today is whether a given text was used to train a large language model (LLM). A completion test is often employed: check if the LLM completes a sufficiently complex text. This, however, requires a ground-truth definition of membership; most commonly, it is defined as a member based on the n-gram overlap between the target text and any text in the dataset. In this work, we demonstrate that this n-gram based membership definition can be effectively gamed. We study scenarios where sequences are non-members for a given n and we find that completion tests still succeed. We find many natural cases of this phenomenon by retraining LLMs from scratch after removing all training samples that were completed; these cases include exact duplicates, near-duplicates, and even short overlaps. They showcase that it is difficult to find a single viable choice of n for membership definitions. Using these insights, we design adversarial datasets that can cause a given target sequence to be completed without containing it, for any reasonable choice of n. Our findings highlight the inadequacy of n-gram membership, suggesting membership definitions fail to account for auxiliary information available to the training algorithm.