# Pandora's White-Box: Precise Training Data Detection and Extraction in Large Language Models

arXiv: https://arxiv.org/abs/2402.17012 · HTML: https://mlsanity.com/paper/2402.17012/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 18 references that are in the map.
- First posted on arXiv 26 Feb 2024.

## Builds on (most cited first, 11 of 11)

- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Quantifying Memorization Across Neural Language Models](https://mlsanity.com/paper/2202.07646/) (2022)
- [Scalable Extraction of Training Data from (Production) Language Models](https://mlsanity.com/paper/2311.17035/) (2023)
- [Detecting Pretraining Data from Large Language Models](https://mlsanity.com/paper/2310.16789/) (2023)
- [Membership Inference Attacks against Language Models via Neighbourhood Comparison](https://mlsanity.com/paper/2305.18462/) (2023)
- [Training Data Extraction From Pre-trained Language Models: A Survey](https://mlsanity.com/paper/2305.16157/) (2023)
- [MoPe: Model Perturbation-based Privacy Attacks on Language Models](https://mlsanity.com/paper/2310.14369/) (2023)
- [Quantifying and Analyzing Entity-level Memorization in Large Language Models](https://mlsanity.com/paper/2308.15727/) (2023)
- [How to Combine Membership-Inference Attacks on Multiple Updated Models](https://mlsanity.com/paper/2205.06369/) (2022)

## Built on by (most cited first, 11 of 11)

- [A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deployment](https://mlsanity.com/paper/2504.15585/) (2025)
- [OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metrics](https://mlsanity.com/paper/2506.12618/) (2025)
- [A Survey on Model Extraction Attacks and Defenses for Large Language Models](https://mlsanity.com/paper/2506.22521/) (2025)
- [Constrained Entropic Unlearning: A Primal-Dual Framework for Large Language Models](https://mlsanity.com/paper/2506.05314/) (2025)
- [Automated Detection of Pre-training Text in Black-box LLMs](https://mlsanity.com/paper/2506.19399/) (2025)
- [ISACL: Internal State Analyzer for Copyrighted Training Data Leakage](https://mlsanity.com/paper/2508.17767/) (2025)
- [Pretraining Data Exposure in Large Language Models: A Survey of Membership Inference, Data Contamination, and Security Implications](https://mlsanity.com/paper/2605.26133/) (2026)
- [LLMSurgeon: Diagnosing Data Mixture of Large Language Models](https://mlsanity.com/paper/2605.30348/) (2026)
- [Collaborative LLM Numerical Reasoning with Local Data Protection](https://mlsanity.com/paper/2504.00299/) (2025)
- [Data Security in Large Language Models: Risks, Defense, and Directions](https://mlsanity.com/paper/2508.02312/) (2025)
- [Open-MMUnlearning: Unifying Methods and Evaluation for MLLM Unlearning](https://mlsanity.com/paper/2610.10358/) (2026)

## Abstract (arXiv)

In this paper we develop state-of-the-art privacy attacks against Large Language Models (LLMs), where an adversary with some access to the model tries to learn something about the underlying training data. Our headline results are new membership inference attacks (MIAs) against pretrained LLMs that perform hundreds of times better than baseline attacks, and a pipeline showing that over 50% (!) of the fine-tuning dataset can be extracted from a fine-tuned LLM in natural settings. We consider varying degrees of access to the underlying model, pretraining and fine-tuning data, and both MIAs and training data extraction. For pretraining data, we propose two new MIAs: a supervised neural network classifier that predicts training data membership on the basis of (dimensionality-reduced) model gradients, as well as a variant of this attack that only requires logit access to the model by leveraging recent model-stealing work on LLMs. To our knowledge this is the first MIA that explicitly incorporates model-stealing information. Both attacks outperform existing black-box baselines, and our supervised attack closes the gap between MIA attack success against LLMs and the strongest known attacks for other machine learning models. In fine-tuning, we find that a simple attack based on the ratio of the loss between the base and fine-tuned models is able to achieve near-perfect MIA performance; we then leverage our MIA to extract a large fraction of the fine-tuning dataset from fine-tuned Pythia and Llama models. Our code is available at github.com/safr-ai-lab/pandora-llm.