mlsanityData to 7 Oct 2026

Teach LLMs to Phish: Stealing Private Information from Language Models

arXiv 2403.00871 ↗First posted 1 Mar 2024cs.CR cs.AI cs.CLcited by 29

Not on the map. Why?

Abstract (arXiv)

When large language models are trained on private data, it can be a significant privacy risk for them to memorize and regurgitate sensitive information. In this work, we propose a new practical data extraction attack that we call "neural phishing". This attack enables an adversary to target and extract sensitive or personally identifiable information (PII), e.g., credit card numbers, from a model trained on user data with upwards of 10% attack success rates, at times, as high as 50%. Our attack assumes only that an adversary can insert as few as 10s of benign-appearing sentences into the training dataset using only vague priors on the structure of the user data.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Builds on 46 of its 74 references that are in the map. First posted on arXiv 1 Mar 2024.

Builds on

46 of 74 references.

References · 46

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  4. 2016Pointer Sentinel Mixture Modelscited by 2,992
  5. 2016Deep Learning with Differential Privacycited by 2,682
  6. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  7. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  8. 2020Extracting Training Data from Large Language Modelscited by 2,144
  9. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  10. 2023PaLM 2 Technical Reportcited by 1,154
  11. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  12. 2021Membership Inference Attacks From First Principlescited by 839
34 more
  1. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  2. 2018How To Backdoor Federated Learningcited by 745
  3. 2023Extracting Training Data from Diffusion Modelscited by 723
  4. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  5. 2018Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networkscited by 500
  6. 2020Label-Only Membership Inference Attackscited by 371
  7. 2018Analyzing Federated Learning through an Adversarial Lenscited by 346
  8. 2017Certified Defenses for Data Poisoning Attackscited by 288
  9. 2019Label-Consistent Backdoor Attackscited by 285
  10. 2023Analyzing Leakage of Personally Identifiable Information in Language Modelscited by 278
  11. 2023Poisoning Web-Scale Training Datasets is Practicalcited by 276
  12. 2018Manipulating Machine Learning: Poisoning Attacks and Countermeasures for Regression Learningcited by 219
  13. 2020Auditing Differentially Private Machine Learning: How Private is Private SGD?cited by 213
  14. 2022Are Large Pre-Trained Language Models Leaking Your Personal Information?cited by 195
  15. 2017Towards Poisoning of Deep Learning Algorithms with Back-gradient Optimizationcited by 188
  16. 2023MADLAD-400: A Multilingual And Document-Level Large Audited Datasetcited by 180
  17. 2021Practical and Private (Deep) Learning without Sampling or Shufflingcited by 169
  18. 2020Witches' Brew: Industrial Scale Data Poisoning via Gradient Matchingcited by 156
  19. 2023Emergent and Predictable Memorization in Large Language Modelscited by 147
  20. 2021The Distributed Discrete Gaussian Mechanism for Federated Learning with Secure Aggregationcited by 127
  21. 2022Measuring Forgetting of Memorized Training Examplescited by 118
  22. 2021Adversarial Examples Make Strong Poisonscited by 115
  23. 2022Neurotoxin: Durable Backdoors in Federated Learningcited by 97
  24. 2022Truth Serum: Poisoning Machine Learning Models to Reveal Their Secretscited by 94
  25. 2016Learning from Untrusted Datacited by 90
  26. 2023Federated Learning of Gboard Language Models with Differential Privacycited by 64
  27. 2022Improved Differential Privacy for SGD via Optimal Private Linear Operators on Adaptive Streamscited by 59
  28. 2023(Amplified) Banded Matrix Factorization: A unified approach to private trainingcited by 49
  29. 2022Multi-Epoch Matrix Factorization Mechanisms for Private Machine Learningcited by 48
  30. 2021SparseFed: Mitigating Model Poisoning Attacks in Federated Learning with Sparsificationcited by 46
  31. 2022The Fundamental Price of Secure Aggregation in Differentially Private Federated Learningcited by 39
  32. 2021CaPC Learning: Confidential and Private Collaborative Learningcited by 33
  33. 2023Students Parrot Their Teachers: Membership Inference on Model Distillationcited by 31
  34. 2023Privacy Amplification for Matrix Mechanismscited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 14; 2026: 5.

29 papers.

Sort:

By area:

  1. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 125
  2. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  3. 2024AirGapAgent: Protecting Privacy-Conscious Conversational Agentscited by 90
  4. 2025Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agentscited by 66
  5. 2024LLM-PBE: Assessing Data Privacy in Large Language Modelscited by 38
  6. 2024Large Model Based Agents: State-of-the-Art, Cooperation Paradigms, Security and Privacy, and Future Trendscited by 26
  7. 2025The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMscited by 24
  8. 2024PII-Compass: Guiding LLM training data extraction prompts towards the target PII via groundingcited by 22
  9. 2025Privacy Auditing of Large Language Modelscited by 22
  10. 2024Zero-Shot Detection of LLM-Generated Text using Token Cohesivenesscited by 13
  11. 2024Mitigating Memorization In Language Modelscited by 11
  12. 2025Invisible Tokens, Visible Bills: The Urgent Need to Audit Hidden Operations in Opaque LLM Servicescited by 10
17 more
  1. 2025Privacy Ripple Effects from Adding or Removing Personal Information in Language Model Trainingcited by 9
  2. 2025The Landscape of Memorization in LLMs: Mechanisms, Measurement, and Mitigationcited by 9
  3. 2024TUNI: A Textual Unimodal Detector for Identity Inference in CLIP Modelscited by 8
  4. 2024Unmasking Database Vulnerabilities: Zero-Knowledge Schema Inference Attacks in Text-to-SQL Systemscited by 6
  5. 2025CapBencher: Give Your LLM Benchmark a Built-in Alarm for Test-Set Overfittingcited by 6
  6. 2025Revisiting Privacy, Utility, and Efficiency Trade-offs when Fine-Tuning Large Language Modelscited by 5
  7. 2025Be Cautious When Merging Unfamiliar LLMs: A Phishing Model Capable of Stealing Privacycited by 3
  8. 2025Clone What You Can't Steal: Black-Box LLM Replication via Logit Leakage and Distillationcited by 3
  9. 2024Interacting Large Language Model Agents. Interpretable Models and Social Learningcited by 2
  10. 2025PATCH: Mitigating PII Leakage in Language Models with Privacy-Aware Targeted Circuit PatcHingcited by 2
  11. 2025LLMs Can Get "Brain Rot": A Pilot Study on Twitter/Xcited by 2
  12. 2026RedacBench: Can AI Erase Your Secrets?cited by 1
  13. 2025The End Of Universal Lifelong Identifiers: Identity Systems For The AI Eracited by 0
  14. 2026PersonaTrace: Synthesizing Realistic Digital Footprints with LLM Agentscited by 0
  15. 2026SoK: Colluding Adversaries in Machine Learning Pipelinescited by 0
  16. 2026MineGrad: Gradient Inversion Attacks on LoRA Fine-Tuningcited by 0
  17. 2026The Privacy Fallacy of Crowdsourced Fine-Tuning: Extracting Proprietary Data via Topic-Based Poisoningcited by 0