# PII-Compass: Guiding LLM training data extraction prompts towards the target PII via grounding

arXiv: https://arxiv.org/abs/2407.02943 · HTML: https://mlsanity.com/paper/2407.02943/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 32 references that are in the map.
- First posted on arXiv 3 Jul 2024.

## Builds on (most cited first, 15 of 15)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Analyzing Leakage of Personally Identifiable Information in Language Models](https://mlsanity.com/paper/2302.00539/) (2023)
- [Are Large Pre-Trained Language Models Leaking Your Personal Information?](https://mlsanity.com/paper/2205.12628/) (2022)
- [Bag of Tricks for Training Data Extraction from Language Models](https://mlsanity.com/paper/2302.04460/) (2023)
- [Can Neural Network Memorization Be Localized?](https://mlsanity.com/paper/2307.09542/) (2023)
- [Training Data Extraction From Pre-trained Language Models: A Survey](https://mlsanity.com/paper/2305.16157/) (2023)
- [Controlling the Extraction of Memorized Data from Large Language Models via Prompt-Tuning](https://mlsanity.com/paper/2305.11759/) (2023)
- [Ethicist: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation](https://mlsanity.com/paper/2307.04401/) (2023)
- [Teach LLMs to Phish: Stealing Private Information from Language Models](https://mlsanity.com/paper/2403.00871/) (2024)
- [Quantifying Association Capabilities of Large Language Models and Its Implications on Privacy Leakage](https://mlsanity.com/paper/2305.12707/) (2023)

## Built on by (most cited first, 24 of 24)

- [A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deployment](https://mlsanity.com/paper/2504.15585/) (2025)
- [Measuring memorization in language models via probabilistic extraction](https://mlsanity.com/paper/2410.19482/) (2024)
- [When LLMs Go Online: The Emerging Threat of Web-Enabled LLMs](https://mlsanity.com/paper/2410.14569/) (2024)
- [Evaluating LLM-based Personal Information Extraction and Countermeasures](https://mlsanity.com/paper/2408.07291/) (2024)
- [The Landscape of Memorization in LLMs: Mechanisms, Measurement, and Mitigation](https://mlsanity.com/paper/2507.05578/) (2025)
- [PII-Scope: A Comprehensive Study on Training Data PII Extraction Attacks in LLMs](https://mlsanity.com/paper/2410.06704/) (2024)
- [PrivacyScalpel: Enhancing LLM Privacy via Interpretable Feature Intervention with Sparse Autoencoders](https://mlsanity.com/paper/2503.11232/) (2025)
- [DP-Fusion: Token-Level Differentially Private Inference for Large Language Models](https://mlsanity.com/paper/2507.04531/) (2025)
- [LCGuard: Latent Communication Guard for Safe KV Sharing in Multi-Agent Systems](https://mlsanity.com/paper/2605.22786/) (2026)
- [Unlearned but Not Forgotten: Data Extraction after Exact Unlearning in LLM](https://mlsanity.com/paper/2505.24379/) (2025)
- [PII Jailbreaking in LLMs via Activation Steering Reveals Personal Information Leakage](https://mlsanity.com/paper/2507.02332/) (2025)
- [What Should LLMs Forget? Quantifying Personal Data in LLMs for Right-to-Be-Forgotten Requests](https://mlsanity.com/paper/2507.11128/) (2025)
- [Retracing the Past: LLMs Emit Training Data When They Get Lost](https://mlsanity.com/paper/2511.05518/) (2025)
- [Unintended Memorization of Sensitive Information in Fine-Tuned Language Models](https://mlsanity.com/paper/2601.17480/) (2026)
- [On the Validity of Traditional Vulnerability Scoring Systems for Adversarial Attacks against LLMs](https://mlsanity.com/paper/2412.20087/) (2024)
- [Beyond PII: How Users Attempt to Estimate and Mitigate Implicit LLM Inference](https://mlsanity.com/paper/2509.12152/) (2025)
- [PATCH: Mitigating PII Leakage in Language Models with Privacy-Aware Targeted Circuit PatcHing](https://mlsanity.com/paper/2510.07452/) (2025)
- [Reverse-Engineering Model Editing on Language Models](https://mlsanity.com/paper/2602.10134/) (2026)
- [LACUNA: A Testbed for Evaluating Localization Precision for LLM Unlearning](https://mlsanity.com/paper/2607.02513/) (2026)
- [Discovering Universal Activation Directions for PII Leakage in Language Models](https://mlsanity.com/paper/2602.16980/) (2026)
- [GEP: A GCG-Based method for extracting personally identifiable information from chatbots built on small language models](https://mlsanity.com/paper/2509.21192/) (2025)
- [On the Effectiveness of Membership Inference in Targeted Data Extraction from Large Language Models](https://mlsanity.com/paper/2512.13352/) (2025)
- [A Multi-Head Attention Approach for SLA Compliance Monitoring in Data Centers](https://mlsanity.com/paper/2605.05354/) (2026)
- [Membership Inference in Fine-tuned Diffusion Language Models via Token-level Memorization Asymmetry](https://mlsanity.com/paper/2609.00873/) (2026)

## Abstract (arXiv)

The latest and most impactful advances in large models stem from their increased size. Unfortunately, this translates into an improved memorization capacity, raising data privacy concerns. Specifically, it has been shown that models can output personal identifiable information (PII) contained in their training data. However, reported PIII extraction performance varies widely, and there is no consensus on the optimal methodology to evaluate this risk, resulting in underestimating realistic adversaries. In this work, we empirically demonstrate that it is possible to improve the extractability of PII by over ten-fold by grounding the prefix of the manually constructed extraction prompt with in-domain data. Our approach, PII-Compass, achieves phone number extraction rates of 0.92%, 3.9%, and 6.86% with 1, 128, and 2308 queries, respectively, i.e., the phone number of 1 person in 15 is extractable.