# Extracting memorized pieces of (copyrighted) books from open-weight language models

arXiv: https://arxiv.org/abs/2505.12546 · HTML: https://mlsanity.com/paper/2505.12546/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 May 2025.

## Built on by (most cited first, 30 of 30)

- [Extracting books from production language models](https://mlsanity.com/paper/2601.02671/) (2026)
- [FlexOlmo: Open Language Models for Flexible Data Use](https://mlsanity.com/paper/2507.07024/) (2025)
- [Comparison requires valid measurement: Rethinking attack success rate comparisons in AI red teaming](https://mlsanity.com/paper/2601.18076/) (2026)
- [Extracting alignment data in open models](https://mlsanity.com/paper/2510.18554/) (2025)
- [Beyond Data Privacy: New Privacy Risks for Large Language Models](https://mlsanity.com/paper/2509.14278/) (2025)
- [Antidistillation Fingerprinting](https://mlsanity.com/paper/2602.03812/) (2026)
- [Alignment Whack-a-Mole : Finetuning Activates Verbatim Recall of Copyrighted Books in Large Language Models](https://mlsanity.com/paper/2603.20957/) (2026)
- [Be Careful When Fine-tuning On Open-Source LLMs: Your Fine-tuning Data Could Be Secretly Stolen!](https://mlsanity.com/paper/2505.15656/) (2025)
- [A Law of Data Reconstruction for Random Features (and Beyond)](https://mlsanity.com/paper/2509.22214/) (2025)
- [K2-V2: A 360-Open, Reasoning-Enhanced LLM](https://mlsanity.com/paper/2512.06201/) (2025)
- [Characterizing Memorization in Diffusion Language Models: Generalized Extraction and Sampling Effects](https://mlsanity.com/paper/2603.02333/) (2026)
- [When Reasoning Leaks Membership: Membership Inference Attack on Black-box Large Reasoning Models](https://mlsanity.com/paper/2601.13607/) (2026)
- [AI Fiction in the Wild](https://mlsanity.com/paper/2606.22748/) (2026)
- [Tell, Don't Show: Leveraging Language Models' Abstractive Retellings to Model Literary Themes](https://mlsanity.com/paper/2505.23166/) (2025)
- [Data Compressibility Quantifies LLM Memorization](https://mlsanity.com/paper/2507.06056/) (2025)
- [Copyright Detective: A Forensic System to Evidence LLMs Flickering Copyright Leakage Risks](https://mlsanity.com/paper/2602.05252/) (2026)
- [Estimating near-verbatim extraction risk in language models with decoding-constrained beam search](https://mlsanity.com/paper/2603.24917/) (2026)
- [LLMs Can Leak Training Data But Do They Want To? A Propensity-Aware Evaluation of Memorization in LLMs](https://mlsanity.com/paper/2606.06286/) (2026)
- [AI Playing Business Games: Benchmarking Large Language Models on Managerial Decision-Making in Dynamic Simulations](https://mlsanity.com/paper/2509.26331/) (2025)
- [RapidUn: Influence-Driven Parameter Reweighting for Efficient Large Language Model Unlearning](https://mlsanity.com/paper/2512.04457/) (2025)
- [Towards Privacy-Preserving Code Generation: Differentially Private Code Language Models](https://mlsanity.com/paper/2512.11482/) (2025)
- [We Should Separate Memorization from Copyright](https://mlsanity.com/paper/2602.08632/) (2026)
- [Memorization Dynamics of Fill-in-the-Middle Pretraining](https://mlsanity.com/paper/2605.22981/) (2026)
- [Probing Stylistic Appropriation using Large Language Models: An Evaluation Framework for Copyright Infringement under EU Law](https://mlsanity.com/paper/2606.31250/) (2026)
- [Watermarking for Proprietary Dataset Protection](https://mlsanity.com/paper/2607.00325/) (2026)
- [Extractable Memorization From First Principles](https://mlsanity.com/paper/2607.12649/) (2026)
- [Leak It: Per-Document Extraction Beyond Aggregate Membership Inference](https://mlsanity.com/paper/2608.00144/) (2026)
- [What to Forget in Unlearning? Forget Set Curation for Language Models](https://mlsanity.com/paper/2608.14855/) (2026)
- [A Sharp Transition in Data Reconstruction under Differential Privacy](https://mlsanity.com/paper/2609.37344/) (2026)
- [It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs](https://mlsanity.com/paper/2609.37891/) (2026)

## Abstract (arXiv)

Plaintiffs and defendants in copyright lawsuits over generative AI often make sweeping, opposing claims about the extent to which large language models (LLMs) memorize protected expression from books in their training data. We show that these polarized positions dramatically oversimplify the relationship between memorization and copyright. To do so, we develop a technique to measure memorization of books, which we apply to 200 books and 14 open-weight LLMs. Through over 3000 experiments, we show that memorization varies both by model and book. With respect to our specific extraction methodology, we find that most LLMs do not memorize most books -- either in whole or in part; however, there are notable exceptions. For instance, Llama 3.1 70B entirely memorizes some books, like Harry Potter and the Sorcerer's Stone; memorization is so extensive that one can deterministically extract the whole book almost verbatim using the book's first few words as an initial prompt. We discuss why our results have significant implications for copyright cases, though not ones that unambiguously favor either side.