# Beyond Public Access in LLM Pre-Training Data

arXiv: https://arxiv.org/abs/2505.00020 · HTML: https://mlsanity.com/paper/2505.00020/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Membership inference attacks, one of 12 directions in LLM safety and security.
- Builds on 16 of its 46 references that are in the map.
- First posted on arXiv 24 Apr 2025.

## Placed in

- LLM safety and security / [Membership inference attacks](https://mlsanity.com/safety/membership-inference-attacks/)

## Builds on (most cited first, 16 of 16)

- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [Quantifying Memorization Across Neural Language Models](https://mlsanity.com/paper/2202.07646/) (2022)
- [Detecting Pretraining Data from Large Language Models](https://mlsanity.com/paper/2310.16789/) (2023)
- [Explainability for Large Language Models: A Survey](https://mlsanity.com/paper/2309.01029/) (2023)
- [Studying Large Language Model Generalization with Influence Functions](https://mlsanity.com/paper/2308.03296/) (2023)
- [Membership Inference Attacks against Language Models via Neighbourhood Comparison](https://mlsanity.com/paper/2305.18462/) (2023)
- [Do Membership Inference Attacks Work on Large Language Models?](https://mlsanity.com/paper/2402.07841/) (2024)
- [Min-K%++: Improved Baseline for Detecting Pre-Training Data from Large Language Models](https://mlsanity.com/paper/2404.02936/) (2024)
- [DE-COP: Detecting Copyrighted Content in Language Models Training Data](https://mlsanity.com/paper/2402.09910/) (2024)
- [Blind Baselines Beat Membership Inference Attacks for Foundation Models](https://mlsanity.com/paper/2406.16201/) (2024)
- [Membership Inference Attacks Cannot Prove that a Model Was Trained On Your Data](https://mlsanity.com/paper/2409.19798/) (2024)
- [Consent in Crisis: The Rapid Decline of the AI Data Commons](https://mlsanity.com/paper/2407.14933/) (2024)
- [Copyright Traps for Large Language Models](https://mlsanity.com/paper/2402.09363/) (2024)
- [Undesirable Memorization in Large Language Models: A Survey](https://mlsanity.com/paper/2410.02650/) (2024)
- [Scaling Up Membership Inference: When and How Attacks Succeed on Large Language Models](https://mlsanity.com/paper/2411.00154/) (2024)

## Built on by (most cited first, 3 of 3)

- [The Attribution Crisis in LLM Search Results](https://mlsanity.com/paper/2508.00838/) (2025)
- [Perturb Your Data: Paraphrase-Guided Training Data Watermarking](https://mlsanity.com/paper/2512.17075/) (2025)
- [Detecting Pretraining Data in Large Language Models from a Free-Energy Perspective](https://mlsanity.com/paper/2609.21888/) (2026)

## Abstract (arXiv)

Using a legally obtained dataset of 34 copyrighted O'Reilly Media books, we apply the DE-COP membership inference attack method to investigate whether OpenAI's large language models show recognition of copyrighted content. Our results based on this small sample suggest that GPT-4o, OpenAI's more recent and capable model, exhibits patterns consistent with recognition of pay-walled book content, with an AUROC score of 0.82 (95% bootstrapped CI: 0.60-0.96), though this wide confidence interval reflects substantial uncertainty due to the limited number of books tested. GPT-4o Mini, as a much smaller model, shows little recognition of any O'Reilly Media content with an AUROC score of 0.56 (0.28-0.83) for non-public data. Testing multiple models, with the same cutoff date, provides a partial control for potential language shifts over time that might bias our findings, though differences in model size, architecture, and potentially training data composition limit the strength of this control. These preliminary results underscore the importance of increased corporate transparency regarding pre-training data sources and the development of formal licensing frameworks for AI content training. Our principal contribution is our examination of public and non public data separately.