mlsanityData to 7 Oct 2026

Privacy Backdoors: Enhancing Membership Inference through Poisoning Pre-trained Models

arXiv 2404.01231 ↗First posted 1 Apr 2024cs.CR cs.LGcited by 28

Not on the map. Why?

Abstract (arXiv)

It is commonplace to produce application-specific models by fine-tuning large pre-trained models using a small bespoke dataset. The widespread availability of foundation model checkpoints on the web poses considerable risks, including the vulnerability to backdoor attacks. In this paper, we unveil a new vulnerability: the privacy backdoor attack. This black-box privacy attack aims to amplify the privacy leakage that arises when fine-tuning a model: when a victim fine-tunes a backdoored model, their training data will be leaked at a significantly higher rate than if they had fine-tuned a typical model. We conduct extensive experiments on various datasets and models, including both vision-language models (CLIP) and large language models, demonstrating the broad applicability and effectiveness of such an attack. Additionally, we carry out multiple ablation studies with different fine-tuning methods and inference strategies to thoroughly analyze this new threat. Our findings highlight a critical privacy concern within the machine learning community and call for a reevaluation of safety protocols in the use of open-source pre-trained models.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 43 of its 62 references that are in the map. First posted on arXiv 1 Apr 2024.

Builds on

43 of 62 references.

References · 43

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  7. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  9. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  10. 2020Extracting Training Data from Large Language Modelscited by 2,144
  11. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  12. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
31 more
  1. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  2. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  3. 2017BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chaincited by 1,052
  4. 2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,012
  5. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  6. 2021Membership Inference Attacks From First Principlescited by 839
  7. 2021Robust fine-tuning of zero-shot modelscited by 785
  8. 2023Extracting Training Data from Diffusion Modelscited by 723
  9. 2020Inverting Gradients -- How easy is it to break privacy in federated learning?cited by 707
  10. 2023A Watermark for Large Language Modelscited by 667
  11. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  12. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 478
  13. 2020Label-Only Membership Inference Attackscited by 371
  14. 2021See through Gradients: Image Batch Recovery via GradInversioncited by 320
  15. 2019White-box vs Black-box: Bayes Optimal Strategies for Membership Inferencecited by 265
  16. 2017Regularizing and Optimizing LSTM Language Modelscited by 205
  17. 2023Are Diffusion Models Vulnerable to Membership Inference Attacks?cited by 135
  18. 2021When the Curious Abandon Honesty: Federated Learning Is Not Privatecited by 126
  19. 2021Robbing the Fed: Directly Obtaining Private Data in Federated Learning with Modified Modelscited by 106
  20. 2023NEFTune: Noisy Embeddings Improve Instruction Finetuningcited by 99
  21. 2022Truth Serum: Poisoning Machine Learning Models to Reveal Their Secretscited by 94
  22. 2022Fishing for User Data in Large-Batch Federated Learning via Gradient Magnificationcited by 72
  23. 2023Language Model Inversioncited by 66
  24. 2021Handcrafted Backdoors in Deep Neural Networkscited by 57
  25. 2022Decepticons: Corrupted Transformers Breach Privacy in Federated Learning for Language Modelscited by 52
  26. 2023Privacy Side Channels in Machine Learning Systemscited by 38
  27. 2022Canary in a Coalmine: Better Membership Inference with Ensembled Adversarial Queriescited by 36
  28. 2024PreCurious: How Innocent Pre-Trained Language Models Turn into Privacy Trapscited by 20
  29. 2024Privacy Backdoors: Stealing Data with Corrupted Pretrained Modelscited by 20
  30. 2020Quantifying Membership Inference Vulnerability via Generalization Gap and Other Model Metricscited by 13
  31. 2023Manipulating Transfer Learning for Property Inferencecited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 12; 2026: 6.

28 papers.

Sort:

By area:

  1. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  2. 2024Be like a Goldfish, Don't Memorize! Mitigating Memorization in Generative LLMscited by 50
  3. 2024Feedback-Guided Extraction of Knowledge Base from Retrieval-Augmented LLM Applicationscited by 36
  4. 2025Ten Challenging Problems in Federated Foundation Modelscited by 20
  5. 2024Nearly Tight Black-Box Auditing of Differentially Private Machine Learningcited by 19
  6. 2024Privacy in Fine-tuning Large Language Models: Attacks, Defenses, and Future Directionscited by 14
  7. 2025Are Robust LLM Fingerprints Adversarially Robust?cited by 7
  8. 2024Forget to Flourish: Leveraging Machine-Unlearning on Pretrained Language Models for Privacy Leakagecited by 6
  9. 2025Your Compiler is Backdooring Your Model: Understanding and Exploiting Compilation Inconsistency Vulnerabilities in Deep Learning Compilerscited by 6
  10. 2024Meanings and Feelings of Large Language Models: Observability of Latent States in Generative AIcited by 5
  11. 2025Be Careful When Fine-tuning On Open-Source LLMs: Your Fine-tuning Data Could Be Secretly Stolen!cited by 5
  12. 2025SoK: Data Reconstruction Attacks Against Machine Learning Models: Definition, Metrics, and Benchmarkcited by 5
16 more
  1. 2024Memory Backdoor Attacks on Neural Networkscited by 3
  2. 2025PoisonedParrot: Subtle Data Poisoning Attacks to Elicit Copyright-Infringing Content from Large Language Modelscited by 3
  3. 2025Gradient Inversion Attacks on Parameter-Efficient Fine-Tuningcited by 3
  4. 2025Quantifying Cross-Modality Memorization in Vision-Language Modelscited by 3
  5. 2024Injecting Undetectable Backdoors in Obfuscated Neural Networks and Language Modelscited by 2
  6. 2025How does Bayesian Sampling help Membership Inference Attacks?cited by 2
  7. 2025What Really is a Member? Discrediting Membership Inference via Poisoningcited by 2
  8. 2025Backdoor Unlearning by Linear Task Decompositioncited by 1
  9. 2026AI Security in the Foundation Model Era: A Comprehensive Survey from a Unified Perspectivecited by 1
  10. 2026Pretraining Data Exposure in Large Language Models: A Survey of Membership Inference, Data Contamination, and Security Implicationscited by 1
  11. 2024Hard Work Does Not Always Pay Off: Poisoning Attacks on Neural Architecture Searchcited by 0
  12. 2024Pre-trained Encoder Inference: Revealing Upstream Encoders In Downstream Machine Learning Servicescited by 0
  13. 2026SoK: Enhancing Cryptographic Collaborative Learning with Differential Privacycited by 0
  14. 2026SoK: Colluding Adversaries in Machine Learning Pipelinescited by 0
  15. 2026Certification of Machine Learning Models via Directional Sharpnesscited by 0
  16. 2026MemCatalyst: Amplifying Data Auditing on Vision-Language Models via Data Poisoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Backdoor in Seconds: Unlocking Vulnerabilities in Large Pre-trained Models via Model Editing
  2. 2023Defending Our Privacy With Backdoors
  3. 2025Membership Inference Attacks on Large-Scale Models: A Survey
  4. 2025A Vision-Language Pre-training Model-Guided Approach for Mitigating Backdoor Attacks in Federated Learning
  5. 2026Loss Landscape Poisoning: Targeted Extraction of Unseen Training Data from LLMs
  6. 2026On the Privacy of LLMs: An Ablation Study
  7. 2024Rethinking Backdoor Detection Evaluation for Language Models
  8. 2024Mitigating Backdoor Threats to Large Language Models: Advancement and Challenges