Beyond Public Access in LLM Pre-Training Data
On the map: LLM safety and security → Membership inference attacks
Abstract (arXiv)
Using a legally obtained dataset of 34 copyrighted O'Reilly Media books, we apply the DE-COP membership inference attack method to investigate whether OpenAI's large language models show recognition of copyrighted content. Our results based on this small sample suggest that GPT-4o, OpenAI's more recent and capable model, exhibits patterns consistent with recognition of pay-walled book content, with an AUROC score of 0.82 (95% bootstrapped CI: 0.60-0.96), though this wide confidence interval reflects substantial uncertainty due to the limited number of books tested. GPT-4o Mini, as a much smaller model, shows little recognition of any O'Reilly Media content with an AUROC score of 0.56 (0.28-0.83) for non-public data. Testing multiple models, with the same cutoff date, provides a partial control for potential language shifts over time that might bias our findings, though differences in model size, architecture, and potentially training data composition limit the strength of this control. These preliminary results underscore the importance of increased corporate transparency regarding pre-training data sources and the development of formal licensing frameworks for AI content training. Our principal contribution is our examination of public and non public data separately.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Membership inference attacks, one of 12 directions in LLM safety and security. Builds on 16 of its 46 references that are in the map. First posted on arXiv 24 Apr 2025.
Builds on
16 of 46 references.
Closest ancestors · 5
Same direction, newest first
- 2024Scaling Up Membership Inference: When and How Attacks Succeed on Large Language Models31 Oct
- 2024Membership Inference Attacks Cannot Prove that a Model Was Trained On Your Data29 Sep
- 2024Blind Baselines Beat Membership Inference Attacks for Foundation Models23 Jun
- 2024Do Membership Inference Attacks Work on Large Language Models?12 Feb
- 2023Membership Inference Attacks against Language Models via Neighbourhood Comparison29 May
Other references · 11
- 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
- 2018Model Cards for Model Reportingcited by 1,080
- 2022Quantifying Memorization Across Neural Language Modelscited by 763
- 2023Detecting Pretraining Data from Large Language Modelscited by 387
- 2023Explainability for Large Language Models: A Surveycited by 333
- 2023Studying Large Language Model Generalization with Influence Functionscited by 326
- 2024Min-K%++: Improved Baseline for Detecting Pre-Training Data from Large Language Modelscited by 103
- 2024DE-COP: Detecting Copyrighted Content in Language Models Training Datacited by 74
- 2024Consent in Crisis: The Rapid Decline of the AI Data Commonscited by 46
- 2024Copyright Traps for Large Language Modelscited by 37
- 2024Undesirable Memorization in Large Language Models: A Surveycited by 29
Built on by
Citing papers per year: 2025: 2; 2026: 1.
3 papers.
Similar, not cited
Close in wording, no citation either way
- 2025As If We've Met Before: LLMs Exhibit Certainty in Recognizing Seen Files
- 2025Bridging the Copyright Gap: Do Large Vision-Language Models Recognize and Respect Copyrighted Content?
- 2026On the Evidentiary Limits of Membership Inference for Copyright Auditing
- 2026Extracting books from production language models
- 2024Evaluating Copyright Takedown Methods for Language Models
- 2024Can Watermarking Large Language Models Prevent Copyrighted Text Generation and Hide Training Data?
- 2024Nob-MIAs: Non-biased Membership Inference Attacks Assessment on Large Language Models with Ex-Post Dataset Construction
- 2024Protecting Copyrighted Material with Unique Identifiers in Large Language Model Training