mlsanityData to 7 Oct 2026

Generated Data with Fake Privacy: Hidden Dangers of Fine-tuning Large Language Models on Generated Data

arXiv 2409.11423 ↗First posted 12 Sep 2024cs.CR cs.LGcited by 21

On the map: Synthetic data and self-improvement for LLMs → Privacy-preserving tabular data synthesis

Abstract (arXiv)

Large language models (LLMs) have demonstrated significant success in various domain-specific tasks, with their performance often improving substantially after fine-tuning. However, fine-tuning with real-world data introduces privacy risks. To mitigate these risks, developers increasingly rely on synthetic data generation as an alternative to using real data, as data generated by traditional models is believed to be different from real-world data. However, with the advanced capabilities of LLMs, the distinction between real data and data generated by these models has become nearly indistinguishable. This convergence introduces similar privacy risks for generated data to those associated with real data. Our study investigates whether fine-tuning with LLM-generated data truly enhances privacy or introduces additional privacy risks by examining the structural characteristics of data generated by LLMs, focusing on two primary fine-tuning approaches: supervised fine-tuning (SFT) with unstructured (plain-text) generated data and self-instruct tuning. In the scenario of SFT, the data is put into a particular instruction tuning format used by previous studies. We use Personal Information Identifier (PII) leakage and Membership Inference Attacks (MIAs) on the Pythia Model Suite and Open Pre-trained Transformer (OPT) to measure privacy risks. Notably, after fine-tuning with unstructured generated data, the rate of successful PII extractions for Pythia increased by over 20%, highlighting the potential privacy implications of such approaches. Furthermore, the ROC-AUC score of MIAs for Pythia-6.9b, the second biggest model of the suite, increases over 40% after self-instruct tuning. Our results indicate the potential privacy risks associated with fine-tuning LLMs using generated data, underscoring the need for careful consideration of privacy safeguards in such approaches.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Placed in Privacy-preserving tabular data synthesis, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 41 of its 57 references that are in the map. First posted on arXiv 12 Sep 2024.

Builds on

41 of 57 references.

References · 41

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  6. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  7. 2016Deep Learning with Differential Privacycited by 2,682
  8. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  9. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  10. 2020Extracting Training Data from Large Language Modelscited by 2,144
  11. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  12. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
29 more
  1. 2021Membership Inference Attacks From First Principlescited by 839
  2. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  3. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  4. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  5. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 505
  6. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 457
  7. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 415
  8. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
  9. 2023Detecting Pretraining Data from Large Language Modelscited by 375
  10. 2023LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Modelscited by 328
  11. 2023VeRA: Vector-based Random Matrix Adaptationcited by 294
  12. 2023Analyzing Leakage of Personally Identifiable Information in Language Modelscited by 278
  13. 2022Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Modelscited by 218
  14. 2021When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Datasetcited by 199
  15. 2022Are Large Pre-Trained Language Models Leaking Your Personal Information?cited by 195
  16. 2024Do Membership Inference Attacks Work on Large Language Models?cited by 188
  17. 2023MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbotscited by 172
  18. 2022Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attackscited by 168
  19. 2023Does Synthetic Data Generation of LLMs Help Clinical Text Mining?cited by 151
  20. 2024AdvPrompter: Fast Adaptive Adversarial Prompting for LLMscited by 146
  21. 2022Privately Fine-Tuning Large Language Models with Differential Privacycited by 39
  22. 2023The Janus Interface: How Fine-Tuning in Large Language Models Amplifies the Privacy Riskscited by 35
  23. 2024MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuningcited by 31
  24. 2024Data Augmentation using Large Language Models: Data Perspectives, Learning Paradigms and Challengescited by 30
  25. 2022On the Privacy Effect of Data Enhancement via the Lens of Memorizationcited by 14
  26. 2020Selecting Informative Contexts Improves Language Model Finetuningcited by 11
  27. 2023Fine-tuning Language Models with Generative Adversarial Reward Modellingcited by 8
  28. 2024Uncovering Latent Memories: Assessing Data Leakage and Memorization Patterns in Frontier AI Modelscited by 4
  29. 2014A Generalized Language Model as the Combination of Skipped n-grams and Modified Kneser-Ney Smoothingcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 14.

21 papers.

Sort:

By area:

  1. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  2. 2025SaLoRA: Safety-Alignment Preserved Low-Rank Adaptationcited by 39
  3. 2025JADES: A Universal Framework for Jailbreak Assessment via Decompositional Scoringcited by 12
  4. 2025A Guardrail for Safety Preservation: When Safety-Sensitive Subspace Meets Harmful-Resistant Null-Spacecited by 9
  5. 2025FLAME-MoE: A Transparent End-to-End Research Platform for Mixture-of-Experts Language Modelscited by 4
  6. 2026Benchmark of Benchmarks: Unpacking Influence and Code Repository Quality in LLM Safety Benchmarkscited by 3
  7. 2025Direct Token Optimization: A Self-contained Approach to Large Language Model Unlearningcited by 2
  8. 2026Finding and Reactivating Post-Trained LLMs' Hidden Safety Mechanismscited by 2
  9. 2026Private Seeds, Public LLMs: Realistic and Privacy-Preserving Synthetic Data Generationcited by 2
  10. 2025Exploring Membership Inference Vulnerabilities in Clinical Large Language Modelscited by 1
  11. 2026Pruning Unsafe Tickets: A Resource-Efficient Framework for Safer and More Robust LLMscited by 1
  12. 2026PPU-Bench:Real World Benchmark for Personalized Partial Unlearning in Vision Language Modelscited by 1
9 more
  1. 2026TrustLDM: Benchmarking Trustworthiness in Language Diffusion Modelscited by 1
  2. 2026Provably Protecting Fine-Tuned LLMs from Training Data Extraction while Preserving Utilitycited by 0
  3. 2026SecureGate: Learning When to Reveal PII Safely via Token-Gated Dual-Adapters for Federated LLMscited by 0
  4. 2026Behavioral Canaries: Auditing Private Retrieved Context Usage in RL Fine-Tuningcited by 0
  5. 2026Reconstruction of Personally Identifiable Information from Proprietary Data in Supervised Fine-Tuned Modelscited by 0
  6. 2026Privacy Leakage in Federated Learning in Radiology Reports: A Comparative Evaluation of Tokenizer-Driven Privacy Riskscited by 0
  7. 2026Position: Privacy Is a Claim, Not a Property of Synthetic Datacited by 0
  8. 2026An Empirical Measurement of Jailbreaking Evaluatorscited by 0
  9. 2026SparLeak: Privacy Leakage from Sparse Attention in LLM Inference on Shared GPUscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025SoK: The Privacy Paradox of Large Language Models: Advancements, Privacy Risks, and Mitigation
  2. 2024SafeSynthDP: Leveraging Large Language Models for Privacy-Preserving Synthetic Data Generation Using Differential Privacy
  3. 2024Privacy-Preserving Instructions for Aligning Large Language Models
  4. 2025The Canary's Echo: Auditing Privacy Risks of LLM-Generated Synthetic Text
  5. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly
  6. 2026Privacy Collapse: Benign Fine-Tuning Can Break Contextual Privacy in Language Models
  7. 2025RewardDS: Privacy-Preserving Fine-Tuning for Large Language Models via Reward Driven Data Synthesis
  8. 2024Understanding Privacy Risks of Embeddings Induced by Large Language Models