# LLM-PBE: Assessing Data Privacy in Large Language Models

arXiv: https://arxiv.org/abs/2408.12787 · HTML: https://mlsanity.com/paper/2408.12787/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Builds on 72 of its 129 references that are in the map.
- First posted on arXiv 23 Aug 2024.

## Builds on (most cited first, 30 of 72)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Jailbroken: How Does LLM Safety Training Fail?](https://mlsanity.com/paper/2307.02483/) (2023)
- [Jailbreaking Black Box Large Language Models in Twenty Queries](https://mlsanity.com/paper/2310.08419/) (2023)
- [Machine Unlearning](https://mlsanity.com/paper/1912.03817/) (2019)
- [Membership Inference Attacks From First Principles](https://mlsanity.com/paper/2112.03570/) (2021)
- [The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks](https://mlsanity.com/paper/1802.08232/) (2018)
- [Ignore Previous Prompt: Attack Techniques For Language Models](https://mlsanity.com/paper/2211.09527/) (2022)
- [Exploiting Unintended Feature Leakage in Collaborative Learning](https://mlsanity.com/paper/1805.04049/) (2018)
- [DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models](https://mlsanity.com/paper/2306.11698/) (2023)
- [What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation](https://mlsanity.com/paper/2008.03703/) (2020)
- [Knowledge Unlearning for Mitigating Privacy Risks in Language Models](https://mlsanity.com/paper/2210.01504/) (2022)
- [Scalable Extraction of Training Data from (Production) Language Models](https://mlsanity.com/paper/2311.17035/) (2023)
- [Large Language Models Can Be Strong Differentially Private Learners](https://mlsanity.com/paper/2110.05679/) (2021)
- [Differentially Private Fine-tuning of Language Models](https://mlsanity.com/paper/2110.06500/) (2021)
- [Explaining Neural Scaling Laws](https://mlsanity.com/paper/2102.06701/) (2021)
- [Multi-step Jailbreaking Privacy Attacks on ChatGPT](https://mlsanity.com/paper/2304.05197/) (2023)
- [Deduplicating Training Data Mitigates Privacy Risks in Language Models](https://mlsanity.com/paper/2202.06539/) (2022)
- [Analyzing Leakage of Personally Identifiable Information in Language Models](https://mlsanity.com/paper/2302.00539/) (2023)
- [Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks](https://mlsanity.com/paper/2302.05733/) (2023)

## Built on by (most cited first, 30 of 41)

- [Understanding World or Predicting Future? A Comprehensive Survey of World Models](https://mlsanity.com/paper/2411.14499/) (2024)
- [A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deployment](https://mlsanity.com/paper/2504.15585/) (2025)
- [The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMs](https://mlsanity.com/paper/2506.11094/) (2025)
- [When LLMs Go Online: The Emerging Threat of Web-Enabled LLMs](https://mlsanity.com/paper/2410.14569/) (2024)
- [A Survey on Privacy Risks and Protection in Large Language Models](https://mlsanity.com/paper/2505.01976/) (2025)
- [Can Textual Gradient Work in Federated Learning?](https://mlsanity.com/paper/2502.19980/) (2025)
- [Membership Inference Attacks as Privacy Tools: Reliability, Disparity and Ensemble](https://mlsanity.com/paper/2506.13972/) (2025)
- [The Landscape of Memorization in LLMs: Mechanisms, Measurement, and Mitigation](https://mlsanity.com/paper/2507.05578/) (2025)
- [Underestimated Privacy Risks for Minority Populations in Large Language Model Unlearning](https://mlsanity.com/paper/2412.08559/) (2024)
- [IP Leakage Attacks Targeting LLM-Based Multi-Agent Systems](https://mlsanity.com/paper/2505.12442/) (2025)
- [PII-Scope: A Comprehensive Study on Training Data PII Extraction Attacks in LLMs](https://mlsanity.com/paper/2410.06704/) (2024)
- [PANORAMA: A synthetic PII-laced dataset for studying sensitive data memorization in LLMs](https://mlsanity.com/paper/2505.12238/) (2025)
- [DP-Fusion: Token-Level Differentially Private Inference for Large Language Models](https://mlsanity.com/paper/2507.04531/) (2025)
- [Beyond Data Privacy: New Privacy Risks for Large Language Models](https://mlsanity.com/paper/2509.14278/) (2025)
- [Position: Weight Space Should Be a First-Class Generative AI Modality](https://mlsanity.com/paper/2605.18632/) (2026)
- [PII Jailbreaking in LLMs via Activation Steering Reveals Personal Information Leakage](https://mlsanity.com/paper/2507.02332/) (2025)
- [Long Context, Less Focus: A Scaling Gap in LLMs Revealed through Privacy and Personalization](https://mlsanity.com/paper/2602.15028/) (2026)
- [POLAR-Bench: A Diagnostic Benchmark for Privacy-Utility Trade-offs in LLM Agents](https://mlsanity.com/paper/2605.19127/) (2026)
- [Multi-PA: A Multi-perspective Benchmark on Privacy Assessment for Large Vision-Language Models](https://mlsanity.com/paper/2412.19496/) (2024)
- [Less or More: Towards Glanceable Explanations for LLM Recommendations Using Ultra-Small Devices](https://mlsanity.com/paper/2502.19410/) (2025)
- [Enterprise AI Must Enforce Participant-Aware Access Control](https://mlsanity.com/paper/2509.14608/) (2025)
- [ACAI for SBOs: AI Co-creation for Advertising and Inspiration for Small Business Owners](https://mlsanity.com/paper/2503.06729/) (2025)
- [Detecting Instruction Fine-tuning Attacks using Influence Function](https://mlsanity.com/paper/2504.09026/) (2025)
- [Beyond PII: How Users Attempt to Estimate and Mitigate Implicit LLM Inference](https://mlsanity.com/paper/2509.12152/) (2025)
- [A Hierarchical Imprecise Probability Approach to Reliability Assessment of Large Language Models](https://mlsanity.com/paper/2511.00527/) (2025)
- [TrustLDM: Benchmarking Trustworthiness in Language Diffusion Models](https://mlsanity.com/paper/2606.00023/) (2026)
- [Bridging Today and the Future of Humanity: AI Safety in 2024 and Beyond](https://mlsanity.com/paper/2410.18114/) (2024)
- [Explore Activation Sparsity in Recurrent LLMs for Energy-Efficient Neuromorphic Computing](https://mlsanity.com/paper/2501.16337/) (2025)
- [PrivacyXray: Detecting Privacy Breaches in LLMs through Semantic Consistency and Probability Certainty](https://mlsanity.com/paper/2506.19563/) (2025)
- [Towards Privacy-Preserving and Personalized Smart Homes via Tailored Small Language Models](https://mlsanity.com/paper/2507.08878/) (2025)

## Abstract (arXiv)

Large Language Models (LLMs) have become integral to numerous domains, significantly advancing applications in data management, mining, and analysis. Their profound capabilities in processing and interpreting complex language data, however, bring to light pressing concerns regarding data privacy, especially the risk of unintentional training data leakage. Despite the critical nature of this issue, there has been no existing literature to offer a comprehensive assessment of data privacy risks in LLMs. Addressing this gap, our paper introduces LLM-PBE, a toolkit crafted specifically for the systematic evaluation of data privacy risks in LLMs. LLM-PBE is designed to analyze privacy across the entire lifecycle of LLMs, incorporating diverse attack and defense strategies, and handling various data types and metrics. Through detailed experimentation with multiple LLMs, LLM-PBE facilitates an in-depth exploration of data privacy concerns, shedding light on influential factors such as model size, data characteristics, and evolving temporal dimensions. This study not only enriches the understanding of privacy issues in LLMs but also serves as a vital resource for future research in the field. Aimed at enhancing the breadth of knowledge in this area, the findings, resources, and our full technical report are made available at https://llm-pbe.github.io/, providing an open platform for academic and practical advancements in LLM privacy assessment.