# Extractable Memorization From First Principles

arXiv: https://arxiv.org/abs/2607.12649 · HTML: https://mlsanity.com/paper/2607.12649/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 33 of its 51 references that are in the map.
- First posted on arXiv 14 Jul 2026.

## Builds on (most cited first, 30 of 33)

- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Membership Inference Attacks From First Principles](https://mlsanity.com/paper/2112.03570/) (2021)
- [The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks](https://mlsanity.com/paper/1802.08232/) (2018)
- [Quantifying Memorization Across Neural Language Models](https://mlsanity.com/paper/2202.07646/) (2022)
- [Deduplicating Training Data Makes Language Models Better](https://mlsanity.com/paper/2107.06499/) (2021)
- [Does Learning Require Memorization? A Short Tale about a Long Tail](https://mlsanity.com/paper/1906.05271/) (2019)
- [Scalable Extraction of Training Data from (Production) Language Models](https://mlsanity.com/paper/2311.17035/) (2023)
- [What Does it Mean for a Language Model to Preserve Privacy?](https://mlsanity.com/paper/2202.05520/) (2022)
- [Counterfactual Memorization in Neural Language Models](https://mlsanity.com/paper/2112.12938/) (2021)
- [Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4](https://mlsanity.com/paper/2305.00118/) (2023)
- [LLM Dataset Inference: Did you train on my dataset?](https://mlsanity.com/paper/2406.06443/) (2024)
- [Copyright Violations and Large Language Models](https://mlsanity.com/paper/2310.13771/) (2023)
- [Preventing Verbatim Memorization in Language Models Gives a False Sense of Privacy](https://mlsanity.com/paper/2210.17546/) (2022)
- [Rethinking LLM Memorization through the Lens of Adversarial Compression](https://mlsanity.com/paper/2404.15146/) (2024)
- [Talkin' 'Bout AI Generation: Copyright and the Generative-AI Supply Chain](https://mlsanity.com/paper/2309.08133/) (2023)
- [Measuring memorization in language models via probabilistic extraction](https://mlsanity.com/paper/2410.19482/) (2024)
- [Extracting memorized pieces of (copyrighted) books from open-weight language models](https://mlsanity.com/paper/2505.12546/) (2025)
- [Exploring the limits of strong membership inference attacks on large language models](https://mlsanity.com/paper/2505.18773/) (2025)
- [The Files are in the Computer: On Copyright, Memorization, and Generative AI](https://mlsanity.com/paper/2404.12590/) (2024)
- [Language Models May Verbatim Complete Text They Were Not Explicitly Trained On](https://mlsanity.com/paper/2503.17514/) (2025)
- [Extracting books from production language models](https://mlsanity.com/paper/2601.02671/) (2026)
- [Comparison requires valid measurement: Rethinking attack success rate comparisons in AI red teaming](https://mlsanity.com/paper/2601.18076/) (2026)
- [Report of the 1st Workshop on Generative AI and Law](https://mlsanity.com/paper/2311.06477/) (2023)
- [Extracting alignment data in open models](https://mlsanity.com/paper/2510.18554/) (2025)
- [Alignment Whack-a-Mole : Finetuning Activates Verbatim Recall of Copyrighted Books in Large Language Models](https://mlsanity.com/paper/2603.20957/) (2026)
- [Machine Learners Should Acknowledge the Legal Implications of Large Language Models as Personal Data](https://mlsanity.com/paper/2503.01630/) (2025)
- [Revisiting the LiRA Membership Inference Attack Under Realistic Assumptions](https://mlsanity.com/paper/2603.07567/) (2026)

## Abstract (arXiv)

Recent work on extractable memorization in language models suffers from two contrasting validity problems. Some studies overstate extraction, for example, by using sequences too short to distinguish memorization from predictability. Others imply that extraction is unreliable evidence because models can reproduce real-world text they weren't explicitly trained on. Both overlook what makes a valid extraction claim: the model must generate a training sequence with high enough probability to indicate memorization. To determine what's high enough, one has to perform a matched comparison, measuring generation probabilities of training and comparable non-training sequences. Since non-training sequences can't have been memorized, their probabilities are a baseline for predictability; exceeding this baseline is evidence of memorization. We formalize matched comparisons with (1) a conformal test calibrated to a chosen false-positive rate when sequences are sampled from populations, and (2) a single-document census that calibrates against a matched non-training document. Matched comparisons enable rigorous, calibrated memorization claims and clarify where prior setups have validity issues. On Wikipedia, OLMo 2 32B reproduces non-training 10-token suffixes roughly 24% as often as training ones: that share reflects false positives, not memorization. For Llama 3.1 70B on books, calibrated census thresholds reach as low as 10^(-27), supporting memorization claims for sequences no feasible sampling budget would extract. We therefore refine "extractable memorization" to require both a valid memorization claim and near-certain generation within a realistic budget.