mlsanityData to 7 Oct 2026

Truth Serum: Poisoning Machine Learning Models to Reveal Their Secrets

arXiv 2204.00032 ↗First posted 31 Mar 2022cs.CR cs.LG stat.MLcited by 94

Not on the map. Why?

Cited by 33 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce a new class of attacks on machine learning models. We show that an adversary who can poison a training dataset can cause models trained on this dataset to leak significant private details of training points belonging to other parties. Our active inference attacks connect two independent lines of work targeting the integrity and privacy of machine learning training data. Our attacks are effective across membership inference, attribute inference, and data extraction. For example, our targeted attacks can poison <0.1% of the training dataset to boost the performance of inference attacks by 1 to 2 orders of magnitude. Further, an adversary who controls a significant fraction of the training data (e.g., 50%) can launch untargeted attacks that enable 8x more precise inference on all other users' otherwise-private data points. Our results cast doubts on the relevance of cryptographic privacy guarantees in multiparty computation protocols for machine learning, if parties can arbitrarily select their share of training data.

Key facts. Cited by 94 papers in the map (data through 7 Oct 2026). Builds on 51 of its 81 references that are in the map. First posted on arXiv 31 Mar 2022.

Builds on

51 of 81 references.

References · 51

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  3. 2016Wide Residual Networkscited by 3,057
  4. 2019Advances and Open Problems in Federated Learningcited by 3,021
  5. 2016Pointer Sentinel Mixture Modelscited by 2,992
  6. 2016Deep Learning with Differential Privacycited by 2,682
  7. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  8. 2020Extracting Training Data from Large Language Modelscited by 2,144
  9. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  10. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  11. 2021Membership Inference Attacks From First Principlescited by 839
  12. 2022Quantifying Memorization Across Neural Language Modelscited by 755
38 more
  1. 2018How To Backdoor Federated Learningcited by 745
  2. 2018Exploiting Unintended Feature Leakage in Collaborative Learningcited by 509
  3. 2018Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networkscited by 500
  4. 2018Spectral Signatures in Backdoor Attackscited by 457
  5. 2019Does Learning Require Memorization? A Short Tale about a Long Tailcited by 452
  6. 2018Analyzing Federated Learning through an Adversarial Lenscited by 346
  7. 2017Deep Models Under the GAN: Information Leakage from Collaborative Deep Learningcited by 326
  8. 2019Label-Consistent Backdoor Attackscited by 285
  9. 2019White-box vs Black-box: Bayes Optimal Strategies for Membership Inferencecited by 265
  10. 2021Enhanced Membership Inference Attacks against Machine Learning Modelscited by 237
  11. 2018Manipulating Machine Learning: Poisoning Attacks and Countermeasures for Regression Learningcited by 219
  12. 2020Auditing Differentially Private Machine Learning: How Private is Private SGD?cited by 213
  13. 2021Adversary Instantiation: Lower Bounds for Differentially Private Machine Learningcited by 193
  14. 2017Towards Poisoning of Deep Learning Algorithms with Back-gradient Optimizationcited by 188
  15. 2019MemGuard: Defending against Black-Box Membership Inference Attacks via Adversarial Examplescited by 183
  16. 2018Machine Learning with Membership Privacy using Adversarial Regularizationcited by 177
  17. 2020Blind Backdoors in Deep Learning Modelscited by 174
  18. 2022Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attackscited by 168
  19. 2017Machine Learning Models that Remember Too Muchcited by 165
  20. 2020Witches' Brew: Industrial Scale Data Poisoning via Gradient Matchingcited by 156
  21. 2017Neural Trojanscited by 151
  22. 2021On the Importance of Difficulty Calibration in Membership Inference Attackscited by 137
  23. 2019Transferable Clean-Label Poisoning Attacks on Deep Neural Netscited by 129
  24. 2021When the Curious Abandon Honesty: Federated Learning Is Not Privatecited by 126
  25. 2018Sever: A Robust Meta-Algorithm for Stochastic Optimizationcited by 123
  26. 2021Adversarial Examples Make Strong Poisonscited by 115
  27. 2020You Autocomplete Me: Poisoning Vulnerabilities in Neural Code Completioncited by 112
  28. 2020Revisiting Membership Inference Under Realistic Assumptionscited by 108
  29. 2016Learning from Untrusted Datacited by 90
  30. 2019Analyzing Information Leakage of Updates to Natural Language Modelscited by 89
  31. 2018Technical Report: When Does Machine Learning FAIL? Generalized Transferability for Evasion and Poisoning Attackscited by 78
  32. 2022Fishing for User Data in Large-Batch Federated Learning via Gradient Magnificationcited by 72
  33. 2020Training Production Language Models without Memorizing User Datacited by 68
  34. 2021Property Inference From Poisoningcited by 56
  35. 2022Decepticons: Corrupted Transformers Breach Privacy in Federated Learning for Language Modelscited by 52
  36. 2021Antipodes of Label Differential Privacy: PATE and ALIBIcited by 50
  37. 2020Unadversarial Examples: Designing Objects for Robust Visioncited by 45
  38. 2022Are Your Sensitive Attributes Private? Novel Model Inversion Attribute Inference Attacks on Classification Modelscited by 42

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 13; 2023: 28; 2024: 25; 2025: 15; 2026: 13.

94 papers.

Sort:

By area:

  1. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 457
  2. 2023Poisoning Web-Scale Training Datasets is Practicalcited by 276
  3. 2022The Privacy Onion Effect: Memorization is Relativecited by 128
  4. 2022Measuring Forgetting of Memorized Training Examplescited by 118
  5. 2023On the Exploitability of Instruction Tuningcited by 110
  6. 2024JailbreakRadar: Comprehensive Assessment of Jailbreak Attacks Against LLMscited by 87
  7. 2023Membership Inference Attacks against Diffusion Modelscited by 84
  8. 2024Don't Listen To Me: Understanding and Exploring Jailbreak Prompts of Large Language Modelscited by 77
  9. 2023Differentially Private Synthetic Data via Foundation Model APIs 1: Imagescited by 63
  10. 2022Rickrolling the Artist: Injecting Backdoors into Text Encoders for Text-to-Image Synthesiscited by 59
  11. 2022SoK: Let the Privacy Games Begin! A Unified Treatment of Data Inference Privacy in Machine Learningcited by 47
  12. 2023Privacy Side Channels in Machine Learning Systemscited by 38
38 more
  1. 2024Evaluations of Machine Learning Privacy Defenses are Misleadingcited by 35
  2. 2023Students Parrot Their Teachers: Membership Inference on Model Distillationcited by 31
  3. 2022SNAP: Efficient Extraction of Private Properties with Poisoningcited by 29
  4. 2022Amplifying Membership Exposure via Data Poisoningcited by 29
  5. 2024Teach LLMs to Phish: Stealing Private Information from Language Modelscited by 29
  6. 2024Privacy Backdoors: Enhancing Membership Inference through Poisoning Pre-trained Modelscited by 28
  7. 2023Overconfidence is a Dangerous Thing: Mitigating Membership Inference Attacks by Enforcing Less Confident Predictioncited by 26
  8. 2022On the (In)security of Peer-to-Peer Decentralized Machine Learningcited by 23
  9. 2022Individual Privacy Accounting for Differentially Private Stochastic Gradient Descentcited by 20
  10. 2024Privacy Backdoors: Stealing Data with Corrupted Pretrained Modelscited by 20
  11. 2024Noisy Neighbors: Efficient membership inference attacks against LLMscited by 20
  12. 2023Unleashing the Power of Randomization in Auditing Differentially Private MLcited by 19
  13. 2024Learning to Poison Large Language Models for Downstream Manipulationcited by 15
  14. 2023TMI! Finetuned Models Leak Private Information from their Pretraining Datacited by 13
  15. 2024Memorized Images in Diffusion Models share a Subspace that can be Located and Deletedcited by 13
  16. 2022Conflicting Interactions Among Protection Mechanisms for Machine Learning Modelscited by 12
  17. 2023Uncovering Adversarial Risks of Test-Time Adaptationcited by 12
  18. 2022Data Isotopes for Data Provenance in DNNscited by 11
  19. 2024Safety in Graph Machine Learning: Threats and Safeguardscited by 11
  20. 2023GraphGuard: Detecting and Counteracting Training Data Misuse in Graph Neural Networkscited by 8
  21. 2024Improving Robustness to Model Inversion Attacks via Sparse Coding Architecturescited by 8
  22. 2024A Survey on Responsible Generative AI: What to Generate and What Notcited by 8
  23. 2025Rectifying Privacy and Efficacy Measurements in Machine Unlearning: A New Inference Attack Perspectivecited by 8
  24. 2022Combing for Credentials: Active Pattern Extraction from Smart Replycited by 7
  25. 2024Efficiently Computing Similarities to Private Datasetscited by 7
  26. 2023A Note On Interpreting Canary Exposurecited by 6
  27. 2023Self-Deception: Reverse Penetrating the Semantic Firewall of Large Language Modelscited by 6
  28. 2023SoK: Unintended Interactions among Machine Learning Defenses and Riskscited by 6
  29. 2024A Method to Facilitate Membership Inference Attacks in Deep Learning Modelscited by 6
  30. 2024Forget to Flourish: Leveraging Machine-Unlearning on Pretrained Language Models for Privacy Leakagecited by 6
  31. 2025Trustworthy Machine Learning via Memorization and the Granular Long-Tail: A Survey on Interactions, Tradeoffs, and Beyondcited by 6
  32. 2025Membership Inference Attacks on Sequence Modelscited by 6
  33. 2025Malice in Agentland: Down the Rabbit Hole of Backdoors in the AI Supply Chaincited by 6
  34. 2022SafeNet: The Unreasonable Effectiveness of Ensembles in Private Collaborative Learningcited by 5
  35. 2023VertexSerum: Poisoning Graph Neural Networks for Link Inferencecited by 5
  36. 2023Unintended Memorization in Large ASR Models, and How to Mitigate Itcited by 5
  37. 2024Rethinking the impact of noisy labels in graph classification: A utility and privacy perspectivecited by 5
  38. 2024Understanding Data Importance in Machine Learning Attacks: Does Valuable Data Pose Greater Harm?cited by 5

50 most cited of 94 All 94 (JSON)