mlsanityData to 14 Oct 2026

PII-Compass: Guiding LLM training data extraction prompts towards the target PII via grounding

arXiv 2407.02943 ↗First posted 3 Jul 2024cs.CR cs.AI cs.CLcited by 24

Not on the map. Why?

Abstract (arXiv)

The latest and most impactful advances in large models stem from their increased size. Unfortunately, this translates into an improved memorization capacity, raising data privacy concerns. Specifically, it has been shown that models can output personal identifiable information (PII) contained in their training data. However, reported PIII extraction performance varies widely, and there is no consensus on the optimal methodology to evaluate this risk, resulting in underestimating realistic adversaries. In this work, we empirically demonstrate that it is possible to improve the extractability of PII by over ten-fold by grounding the prefix of the manually constructed extraction prompt with in-domain data. Our approach, PII-Compass, achieves phone number extraction rates of 0.92%, 3.9%, and 6.86% with 1, 128, and 2308 queries, respectively, i.e., the phone number of 1 person in 15 is extractable.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Builds on 15 of its 32 references that are in the map. First posted on arXiv 3 Jul 2024.

Builds on

15 of 32 references.

References · 15

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  3. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,898
  4. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  5. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  6. 2020Extracting Training Data from Large Language Modelscited by 2,189
  7. 2023Analyzing Leakage of Personally Identifiable Information in Language Modelscited by 285
  8. 2022Are Large Pre-Trained Language Models Leaking Your Personal Information?cited by 200
  9. 2023Bag of Tricks for Training Data Extraction from Language Modelscited by 70
  10. 2023Can Neural Network Memorization Be Localized?cited by 69
  11. 2023Training Data Extraction From Pre-trained Language Models: A Surveycited by 45
  12. 2023Controlling the Extraction of Memorized Data from Large Language Models via Prompt-Tuningcited by 40
3 more
  1. 2023Ethicist: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimationcited by 36
  2. 2024Teach LLMs to Phish: Stealing Private Information from Language Modelscited by 32
  3. 2023Quantifying Association Capabilities of Large Language Models and Its Implications on Privacy Leakagecited by 28

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 12; 2026: 7.

24 papers.

Sort:

By area:

  1. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 116
  2. 2024Measuring memorization in language models via probabilistic extractioncited by 41
  3. 2024When LLMs Go Online: The Emerging Threat of Web-Enabled LLMscited by 24
  4. 2024Evaluating LLM-based Personal Information Extraction and Countermeasurescited by 12
  5. 2025The Landscape of Memorization in LLMs: Mechanisms, Measurement, and Mitigationcited by 10
  6. 2024PII-Scope: A Comprehensive Study on Training Data PII Extraction Attacks in LLMscited by 7
  7. 2025PrivacyScalpel: Enhancing LLM Privacy via Interpretable Feature Intervention with Sparse Autoencoderscited by 6
  8. 2025DP-Fusion: Token-Level Differentially Private Inference for Large Language Modelscited by 6
  9. 2026LCGuard: Latent Communication Guard for Safe KV Sharing in Multi-Agent Systemscited by 6
  10. 2025Unlearned but Not Forgotten: Data Extraction after Exact Unlearning in LLMcited by 5
  11. 2025PII Jailbreaking in LLMs via Activation Steering Reveals Personal Information Leakagecited by 5
  12. 2025What Should LLMs Forget? Quantifying Personal Data in LLMs for Right-to-Be-Forgotten Requestscited by 4
12 more
  1. 2025Retracing the Past: LLMs Emit Training Data When They Get Lostcited by 3
  2. 2026Unintended Memorization of Sensitive Information in Fine-Tuned Language Modelscited by 3
  3. 2024On the Validity of Traditional Vulnerability Scoring Systems for Adversarial Attacks against LLMscited by 2
  4. 2025Beyond PII: How Users Attempt to Estimate and Mitigate Implicit LLM Inferencecited by 2
  5. 2025PATCH: Mitigating PII Leakage in Language Models with Privacy-Aware Targeted Circuit PatcHingcited by 2
  6. 2026Reverse-Engineering Model Editing on Language Modelscited by 2
  7. 2026LACUNA: A Testbed for Evaluating Localization Precision for LLM Unlearningcited by 2
  8. 2026Discovering Universal Activation Directions for PII Leakage in Language Modelscited by 1
  9. 2025GEP: A GCG-Based method for extracting personally identifiable information from chatbots built on small language modelscited by 0
  10. 2025On the Effectiveness of Membership Inference in Targeted Data Extraction from Large Language Modelscited by 0
  11. 2026A Multi-Head Attention Approach for SLA Compliance Monitoring in Data Centerscited by 0
  12. 2026Membership Inference in Fine-tuned Diffusion Language Models via Token-level Memorization Asymmetrycited by 0