mlsanityData to 14 Oct 2026

PreCurious: How Innocent Pre-Trained Language Models Turn into Privacy Traps

arXiv 2403.09562 ↗First posted 14 Mar 2024cs.CRcited by 23

On the map: LLM safety and security → Membership inference attacks

Abstract (arXiv)

The pre-training and fine-tuning paradigm has demonstrated its effectiveness and has become the standard approach for tailoring language models to various tasks. Currently, community-based platforms offer easy access to various pre-trained models, as anyone can publish without strict validation processes. However, a released pre-trained model can be a privacy trap for fine-tuning datasets if it is carefully designed. In this work, we propose PreCurious framework to reveal the new attack surface where the attacker releases the pre-trained model and gets a black-box access to the final fine-tuned model. PreCurious aims to escalate the general privacy risk of both membership inference and data extraction on the fine-tuning dataset. The key intuition behind PreCurious is to manipulate the memorization stage of the pre-trained model and guide fine-tuning with a seemingly legitimate configuration. While empirical and theoretical evidence suggests that parameter-efficient and differentially private fine-tuning techniques can defend against privacy attacks on a fine-tuned model, PreCurious demonstrates the possibility of breaking up this invulnerability in a stealthy manner compared to fine-tuning on a benign pre-trained model. While DP provides some mitigation for membership inference attack, by further leveraging a sanitized dataset, PreCurious demonstrates potential vulnerabilities for targeted data extraction even under differentially private tuning with a strict privacy budget e.g. ε=0.05. Thus, PreCurious raises warnings for users on the potential risks of downloading pre-trained models from unknown sources, relying solely on tutorials or common-sense defenses, and releasing sanitized datasets even after perfect scrubbing.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Placed in Membership inference attacks, one of 12 directions in LLM safety and security. Builds on 43 of its 76 references that are in the map. First posted on arXiv 14 Mar 2024.

Builds on

43 of 76 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Last One Standing: A Comparative Analysis of Security and Privacy of Soft Prompt Tuning, LoRA, and In-Context Learning17 Oct
  2. 2023Membership Inference Attacks against Language Models via Neighbourhood Comparison29 May
  3. 2022Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks8 Mar
  4. 2021Membership Inference Attacks From First Principles7 Dec
  5. 2021Enhanced Membership Inference Attacks against Machine Learning Models18 Nov
  6. 2021On the Importance of Difficulty Calibration in Membership Inference Attacks15 Nov

Other references · 37

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  3. 2016Deep Learning with Differential Privacycited by 2,728
  4. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
  5. 2020Extracting Training Data from Large Language Modelscited by 2,189
  6. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  7. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  8. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  9. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 864
  10. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  11. 2018Exploiting Unintended Feature Leakage in Collaborative Learningcited by 520
  12. 2019ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmissioncited by 493
25 more
  1. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 478
  2. 2020AdapterHub: A Framework for Adapting Transformerscited by 444
  3. 2013DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognitioncited by 424
  4. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 412
  5. 2021Differentially Private Fine-tuning of Language Modelscited by 400
  6. 2020Label-Only Membership Inference Attackscited by 388
  7. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 314
  8. 2019White-box vs Black-box: Bayes Optimal Strategies for Membership Inferencecited by 271
  9. 2019MemGuard: Defending against Black-Box Membership Inference Attacks via Adversarial Examplescited by 197
  10. 2023Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuningcited by 173
  11. 2020Membership Leakage in Label-Only Exposurescited by 168
  12. 2020Revisiting Membership Inference Under Realistic Assumptionscited by 112
  13. 2022Truth Serum: Poisoning Machine Learning Models to Reveal Their Secretscited by 101
  14. 2018Understanding Membership Inferences on Well-Generalized Learning Modelscited by 90
  15. 2020Trojaning Language Models for Fun and Profitcited by 90
  16. 2023Can Neural Network Memorization Be Localized?cited by 69
  17. 2021Property Inference From Poisoningcited by 60
  18. 2022SoK: Let the Privacy Games Begin! A Unified Treatment of Data Inference Privacy in Machine Learningcited by 51
  19. 2022Memorization in NLP Fine-tuning Methodscited by 45
  20. 2022Amplifying Membership Exposure via Data Poisoningcited by 34
  21. 2020Investigating Membership Inference Attacks under Data Dependenciescited by 32
  22. 2023TMI! Finetuned Models Leak Private Information from their Pretraining Datacited by 14
  23. 2023Initialization Matters: Privacy-Utility Analysis of Overparameterized Neural Networkscited by 10
  24. 2023Manipulating Transfer Learning for Property Inferencecited by 8
  25. 2023Information Leakage from Data Updates in Machine Learning Modelscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 8; 2025: 10; 2026: 5.

23 papers.

Sort:

By area:

  1. 2024Privacy Backdoors: Enhancing Membership Inference through Poisoning Pre-trained Modelscited by 30
  2. 2024Privacy Backdoors: Stealing Data with Corrupted Pretrained Modelscited by 22
  3. 2024Nearly Tight Black-Box Auditing of Differentially Private Machine Learningcited by 21
  4. 2025Tokens for Learning, Tokens for Unlearning: Mitigating Membership Inference Attacks in Large Language Models via Dual-Purpose Trainingcited by 15
  5. 2024Privacy in Fine-tuning Large Language Models: Attacks, Defenses, and Future Directionscited by 14
  6. 2025SALAD: Systematic Assessment of Machine Unlearning on LLM-Aided Hardware Designcited by 12
  7. 2025VeriLeaky: Navigating IP Protection vs Utility in Fine-Tuning for LLM-Driven Verilog Codingcited by 11
  8. 2025LoRA-Leak: Membership Inference Attacks Against LoRA Fine-tuned Language Modelscited by 9
  9. 2025Your Compiler is Backdooring Your Model: Understanding and Exploiting Compilation Inconsistency Vulnerabilities in Deep Learning Compilerscited by 7
  10. 2024Forget to Flourish: Leveraging Machine-Unlearning on Pretrained Language Models for Privacy Leakagecited by 6
  11. 2025Are Your LLM-based Text-to-SQL Models Secure? Exploring SQL Injection via Backdoor Attackscited by 6
  12. 2025Be Careful When Fine-tuning On Open-Source LLMs: Your Fine-tuning Data Could Be Secretly Stolen!cited by 5
11 more
  1. 2025SoK: Data Reconstruction Attacks Against Machine Learning Models: Definition, Metrics, and Benchmarkcited by 5
  2. 2025Exploiting Leaderboards for Large-Scale Distribution of Malicious Modelscited by 5
  3. 2024Reconstruction of Differentially Private Text Sanitization via Large Language Modelscited by 4
  4. 2024LoRA Provides Differential Privacy by Design via Random Sketchingcited by 3
  5. 2024ExpShield: Safeguarding Web Text from Unauthorized Crawling and LLM Exploitationcited by 3
  6. 2025Gradient Inversion Attacks on Parameter-Efficient Fine-Tuningcited by 3
  7. 2026Beyond Indistinguishability: Measuring Extraction Risk in LLM APIscited by 1
  8. 2026Privacy Enhanced PEFT: Tensor Train Decomposition Improves Privacy Utility Tradeoffs under DP-SGDcited by 0
  9. 2026Secret Stealing Attacks on Local LLM Fine-Tuning through Supply-Chain Model Code Backdoorscited by 0
  10. 2026MineGrad: Gradient Inversion Attacks on LoRA Fine-Tuningcited by 0
  11. 2026SparLeak: Privacy Leakage from Sparse Attention in LLM Inference on Shared GPUscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024SoK: Reducing the Vulnerability of Fine-tuned Language Models to Membership Inference Attacks
  2. 2024On Active Privacy Auditing in Supervised Fine-tuning for White-Box Language Models
  3. 2024Generated Data with Fake Privacy: Hidden Dangers of Fine-tuning Large Language Models on Generated Data
  4. 2025Can Differentially Private Fine-tuning LLMs Protect Against Privacy Attacks?
  5. 2025Impact of Fine-Tuning Methods on Memorization in Large Language Models
  6. 2026Unintended Memorization of Sensitive Information in Fine-Tuned Language Models
  7. 2024Efficient and Private: Memorisation under differentially private parameter-efficient fine-tuning in language models
  8. 2025A Survey on Privacy Risks and Protection in Large Language Models