mlsanityData to 7 Oct 2026

Privacy Backdoors: Stealing Data with Corrupted Pretrained Models

arXiv 2404.00473 ↗First posted 30 Mar 2024cs.CR cs.LGcited by 20

Not on the map. Why?

Abstract (arXiv)

Practitioners commonly download pretrained machine learning models from open repositories and finetune them to fit specific applications. We show that this practice introduces a new risk of privacy backdoors. By tampering with a pretrained model's weights, an attacker can fully compromise the privacy of the finetuning data. We show how to build privacy backdoors for a variety of models, including transformers, which enable an attacker to reconstruct individual finetuning samples, with a guaranteed success! We further show that backdoored models allow for tight privacy attacks on models trained with differential privacy (DP). The common optimistic practice of training DP models with loose privacy guarantees is thus insecure if the model is not trusted. Overall, our work highlights a crucial and overlooked supply chain attack on machine learning privacy.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 31 of its 45 references that are in the map. First posted on arXiv 30 Mar 2024.

Builds on

31 of 45 references.

References · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  5. 2016Deep Learning with Differential Privacycited by 2,682
  6. 2020Extracting Training Data from Large Language Modelscited by 2,144
  7. 2017BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chaincited by 1,052
  8. 2023Extracting Training Data from Diffusion Modelscited by 723
  9. 2016Stealing Machine Learning Models via Prediction APIscited by 602
  10. 2021Opacus: User-Friendly Differential Privacy Library in PyTorchcited by 375
  11. 2013The Composition Theorem for Differential Privacycited by 290
  12. 2020Differentially Private Learning Needs Better Features (or Much More Data)cited by 233
19 more
  1. 2019Understanding and Improving Layer Normalizationcited by 213
  2. 2020Auditing Differentially Private Machine Learning: How Private is Private SGD?cited by 213
  3. 2019R'enyi Differential Privacy of the Sampled Gaussian Mechanismcited by 199
  4. 2021Adversary Instantiation: Lower Bounds for Differentially Private Machine Learningcited by 193
  5. 2017Machine Learning Models that Remember Too Muchcited by 165
  6. 2021When the Curious Abandon Honesty: Federated Learning Is Not Privatecited by 126
  7. 2023Privacy Auditing with One (1) Training Runcited by 119
  8. 2021Robbing the Fed: Directly Obtaining Private Data in Federated Learning with Modified Modelscited by 106
  9. 2018Privacy Amplification by Iterationcited by 97
  10. 2022Truth Serum: Poisoning Machine Learning Models to Reveal Their Secretscited by 94
  11. 2019Reverse-Engineering Deep ReLU Networkscited by 81
  12. 2022Fishing for User Data in Large-Batch Federated Learning via Gradient Magnificationcited by 72
  13. 2021Differential Privacy Dynamics of Langevin Diffusion and Noisy Gradient Descentcited by 71
  14. 2020Training Production Language Models without Memorizing User Datacited by 68
  15. 2020Cryptanalytic Extraction of Neural Network Modelscited by 65
  16. 2022Decepticons: Corrupted Transformers Breach Privacy in Federated Learning for Language Modelscited by 52
  17. 2023LOKI: Large-scale Data Reconstruction Attack against Federated Learning through Model Manipulationcited by 35
  18. 2024PreCurious: How Innocent Pre-Trained Language Models Turn into Privacy Trapscited by 20
  19. 2023Polynomial Time Cryptanalytic Extraction of Neural Network Modelscited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 8; 2026: 4.

20 papers.

Sort:

By area:

  1. 2024Evaluations of Machine Learning Privacy Defenses are Misleadingcited by 35
  2. 2024Privacy Backdoors: Enhancing Membership Inference through Poisoning Pre-trained Modelscited by 28
  3. 2024Backdoored Retrievers for Prompt Injection Attacks on Retrieval Augmented Generation of Large Language Modelscited by 21
  4. 2024Nearly Tight Black-Box Auditing of Differentially Private Machine Learningcited by 19
  5. 2025A Survey on Model Extraction Attacks and Defenses for Large Language Modelscited by 18
  6. 2024Tighter Privacy Auditing of DP-SGD in the Hidden State Threat Modelcited by 17
  7. 2024Forget to Flourish: Leveraging Machine-Unlearning on Pretrained Language Models for Privacy Leakagecited by 6
  8. 2025Privacy-Aware Detection of Fake Identity Documents: Methodology, Benchmark, and Improved Algorithms (FakeIDet2)cited by 6
  9. 2025Be Careful When Fine-tuning On Open-Source LLMs: Your Fine-tuning Data Could Be Secretly Stolen!cited by 5
  10. 2025The Hitchhiker's Guide to Efficient, End-to-End, and Tight DP Auditingcited by 4
  11. 2025Gradient Inversion Attacks on Parameter-Efficient Fine-Tuningcited by 3
  12. 2025Clone What You Can't Steal: Black-Box LLM Replication via Logit Leakage and Distillationcited by 3
8 more
  1. 2025Tuning without Peeking: Provable Generalization Bounds and Robust LLM Post-Trainingcited by 1
  2. 2025Beyond Training-time Poisoning: Component-level and Post-training Backdoors in Deep Reinforcement Learningcited by 1
  3. 2024Pre-trained Encoder Inference: Revealing Upstream Encoders In Downstream Machine Learning Servicescited by 0
  4. 2024Behavior Backdoor for Deep Learning Modelscited by 0
  5. 2026SoK: Enhancing Cryptographic Collaborative Learning with Differential Privacycited by 0
  6. 2026Secret Stealing Attacks on Local LLM Fine-Tuning through Supply-Chain Model Code Backdoorscited by 0
  7. 2026SoK: Colluding Adversaries in Machine Learning Pipelinescited by 0
  8. 2026MineGrad: Gradient Inversion Attacks on LoRA Fine-Tuningcited by 0