mlsanityData to 7 Oct 2026

Reconstruction of Personally Identifiable Information from Proprietary Data in Supervised Fine-Tuned Models

arXiv 2605.12264 ↗First posted 12 May 2026cs.CR cs.CL cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Supervised Finetuning (SFT) has become one of the primary methods for adapting a large language model (LLM) with extensive pre-trained knowledge to domain-specific, instruction-following tasks. SFT datasets, composed of instruction-response pairs, often include user-provided information that may contain sensitive data such as personally identifiable information (PII), raising privacy concerns. This paper studies the problem of targeted PII reconstruction from models fine- tuned on proprietary SFT data, in which an adversary attempts to recover PII associated with a specific identity. We construct multi-turn, user-centric Q&A datasets in sensitive domains, specifically medical and legal settings, that incorporate PII to enable realistic evaluation of leakage. We then propose COVA, a coverage-aware decoding algorithm for targeted PII reconstruction under prefix-based attacks. Using COVA, we study how the amount of information available about a target user affects the recovery of their PII from SFT models. Across datasets, COVA consistently improves PII reconstruction over baseline decoding methods, and even limited contextual knowledge can substantially increase an adversary's success. Our findings demonstrate that small, proprietary SFT datasets can induce meaningful privacy leakage through the reconstruction of target-PII associations learned during fine-tuning.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 34 of its 61 references that are in the map. First posted on arXiv 12 May 2026.

Builds on

34 of 61 references.

References · 34

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  4. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  5. 2020Extracting Training Data from Large Language Modelscited by 2,144
  6. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  7. 2021Membership Inference Attacks From First Principlescited by 839
  8. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  9. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  10. 2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 317
  11. 2023Analyzing Leakage of Personally Identifiable Information in Language Modelscited by 278
  12. 2023Magicoder: Empowering Code Generation with OSS-Instructcited by 251
22 more
  1. 2023How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Compositioncited by 222
  2. 2019Neural Legal Judgment Prediction in Englishcited by 208
  3. 2024MLLM-Protector: Ensuring MLLM's Safety without Hurting Performancecited by 111
  4. 2023Practical Membership Inference Attacks against Fine-tuned Large Language Models via Self-prompt Calibrationcited by 90
  5. 2021Mitigating Membership Inference Attacks by Self-Distillation Through a Novel Ensemble Architecturecited by 63
  6. 2021Selective Differential Privacy for Language Modelingcited by 61
  7. 2023Exploring Memorization in Fine-tuned Language Modelscited by 44
  8. 2024Trust No Bot: Discovering Personal Disclosures in Human-LLM Conversations in the Wildcited by 43
  9. 2022Membership Inference Attacks and Defenses in Neural Network Pruningcited by 42
  10. 2023The Janus Interface: How Fine-Tuning in Large Language Models Amplifies the Privacy Riskscited by 35
  11. 2024Generated Data with Fake Privacy: Hidden Dangers of Fine-tuning Large Language Models on Generated Datacited by 21
  12. 2024Noisy Neighbors: Efficient membership inference attacks against LLMscited by 20
  13. 2025Exploring the limits of strong membership inference attacks on large language modelscited by 20
  14. 2022Membership Inference Attacks and Generalization: A Causal Perspectivecited by 17
  15. 2025Position: Privacy Is Not Just Memorization!cited by 15
  16. 2026Large-scale online deanonymization with LLMscited by 13
  17. 2022Parameter-Efficient Legal Domain Adaptationcited by 12
  18. 2025Privacy Ripple Effects from Adding or Removing Personal Information in Language Model Trainingcited by 9
  19. 2025Differentiation-Based Extraction of Proprietary Data from Fine-Tuned LLMscited by 6
  20. 2026Unintended Memorization of Sensitive Information in Fine-Tuned Language Modelscited by 3
  21. 2025PATCH: Mitigating PII Leakage in Language Models with Privacy-Aware Targeted Circuit PatcHingcited by 2
  22. 2026CLIOPATRA: Extracting Private Information from LLM Insightscited by 1

Built on by

Not cited yet.