mlsanityData to 7 Oct 2026

Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attacks

arXiv 2309.17410 ↗First posted 29 Sep 2023cs.CL cs.AI cs.LGcited by 156

Not on the map. Why?

Cited by 102 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Pretrained language models sometimes possess knowledge that we do not wish them to, including memorized personal information and knowledge that could be used to harm people. They can also output toxic or harmful text. To mitigate these safety and informational issues, we propose an attack-and-defense framework for studying the task of deleting sensitive information directly from model weights. We study direct edits to model weights because (1) this approach should guarantee that particular deleted information is never extracted by future prompt attacks, and (2) it should protect against whitebox attacks, which is necessary for making claims about safety/privacy in a setting where publicly available model weights could be used to elicit sensitive information. Our threat model assumes that an attack succeeds if the answer to a sensitive question is located among a set of B generated candidates, based on scenarios where the information would be insecure if the answer is among B candidates. Experimentally, we show that even state-of-the-art model editing methods such as ROME struggle to truly delete factual information from models like GPT-J, as our whitebox and blackbox attacks can recover "deleted" information from an edited model 38% of the time. These attacks leverage two key observations: (1) that traces of deleted information can be found in intermediate model hidden states, and (2) that applying an editing method for one question may not delete information across rephrased versions of the question. Finally, we provide new defense methods that protect against some extraction attacks, but we do not find a single universally effective defense method. Our results suggest that truly deleting sensitive information is a tractable but difficult problem, since even relatively low attack success rates have potentially severe societal implications for real-world deployment of language models.

Key facts. Cited by 156 papers in the map (data through 7 Oct 2026). Builds on 49 of its 59 references that are in the map. First posted on arXiv 29 Sep 2023.

Builds on

49 of 59 references.

References · 49

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  6. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  7. 2020Extracting Training Data from Large Language Modelscited by 2,144
  8. 2019Language Models as Knowledge Bases?cited by 1,922
  9. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  10. 2022Editing Models with Task Arithmeticcited by 1,076
  11. 2022Mass-Editing Memory in a Transformercited by 976
  12. 2021Ethical and social risks of harm from Language Modelscited by 957
37 more
  1. 2019How Can We Know What Language Models Know?cited by 953
  2. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  3. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  4. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  5. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  6. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  7. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  8. 2021Editing Factual Knowledge in Language Modelscited by 548
  9. 2023Erasing Concepts from Diffusion Modelscited by 530
  10. 2019Certified Data Removal from Machine Learning Modelscited by 512
  11. 2021Fast Model Editing at Scalecited by 491
  12. 2019On Evaluating Adversarial Robustnesscited by 442
  13. 2020Null It Out: Guarding Protected Attributes by Iterative Nullspace Projectioncited by 442
  14. 2017Zero-Shot Relation Extraction via Reading Comprehensioncited by 425
  15. 2023Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defensecited by 401
  16. 2023Ablating Concepts in Text-to-Image Diffusion Modelscited by 312
  17. 2023Forget-Me-Not: Learning to Forget in Text-to-Image Diffusion Modelscited by 297
  18. 2023Analyzing Leakage of Personally Identifiable Information in Language Modelscited by 278
  19. 2023LEACE: Perfect linear concept erasure in closed formcited by 254
  20. 2023Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Modelscited by 237
  21. 2021Mind the Gap: Assessing Temporal Generalization in Neural Language Modelscited by 226
  22. 2020Modifying Memories in Transformer Modelscited by 222
  23. 2022What Does it Mean for a Language Model to Preserve Privacy?cited by 194
  24. 2023Selective Amnesia: A Continual Learning Approach to Forgetting in Deep Generative Modelscited by 175
  25. 2021Alignment of Language Agentscited by 167
  26. 2022Language Models as Agent Modelscited by 131
  27. 2023Foundation Models and Fair Usecited by 112
  28. 2017Ethical Challenges in Data-Driven Dialogue Systemscited by 86
  29. 2021Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefscited by 79
  30. 2023Use of LLMs for Illicit Purposes: Threats, Prevention Measures, and Vulnerabilitiescited by 76
  31. 2023SILO Language Models: Isolating Legal Risk In a Nonparametric Datastorecited by 71
  32. 2023Right to be Forgotten in the Era of Large Language Models: Implications, Challenges, and Solutionscited by 59
  33. 2023Training Data Extraction From Pre-trained Language Models: A Surveycited by 43
  34. 2023Privacy Side Channels in Machine Learning Systemscited by 38
  35. 2022Gold Doesn't Always Glitter: Spectral Removal of Linear and Nonlinear Guarded Attribute Informationcited by 33
  36. 2022Repairing Neural Networks by Leaving the Right Past Behindcited by 27
  37. 2019Analyzing GDPR Compliance Through the Lens of Privacy Policycited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 60; 2025: 59; 2026: 32.

156 papers.

Sort:

By area:

  1. 2024TOFU: A Task of Fictitious Unlearning for LLMscited by 507
  2. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearningcited by 484
  3. 2023SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generationcited by 376
  4. 2024Agent-SafetyBench: Evaluating the Safety of LLM Agentscited by 280
  5. 2024Rethinking Machine Unlearning for Large Language Modelscited by 247
  6. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 158
  7. 2024Eight Methods to Evaluate Robust Unlearning in LLMscited by 153
  8. 2024SAFREE: Training-Free and Adaptive Guard for Safe Text-to-Image And Video Generationcited by 125
  9. 2024An Adversarial Perspective on Machine Unlearning for AI Safetycited by 113
  10. 2023A Survey of Text Watermarking in the Era of Large Language Modelscited by 102
  11. 2024Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Spacecited by 88
  12. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 86
38 more
  1. 2024Unlearning or Obfuscating? Jogging the Memory of Unlearned LLMs via Benign Relearningcited by 81
  2. 2024RWKU: Benchmarking Real-World Knowledge Unlearning for Large Language Modelscited by 77
  3. 2024Catastrophic Failure of LLM Unlearning via Quantizationcited by 76
  4. 2025Towards LLM Unlearning Resilient to Relearning Attacks: A Sharpness-Aware Minimization Perspective and Beyondcited by 76
  5. 2024LLM Unlearning via Loss Adjustment with Only Forget Datacited by 71
  6. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Modelscited by 67
  7. 2024On the Societal Impact of Open Foundation Modelscited by 64
  8. 2025Rethinking LLM Unlearning Objectives: A Gradient Perspective and Go Beyondcited by 59
  9. 2024Position: LLM Unlearning Benchmarks are Weak Measures of Progresscited by 58
  10. 2024Larimar: Large Language Models with Episodic Memory Controlcited by 53
  11. 2024Rebuilding ROME : Resolving Model Collapse during Sequential Model Editingcited by 50
  12. 2023Privacy in Large Language Models: Attacks, Defenses and Future Directionscited by 48
  13. 2024Evaluating Copyright Takedown Methods for Language Modelscited by 42
  14. 2025Unlearning Isn't Deletion: Investigating Reversibility of Machine Unlearning in LLMscited by 41
  15. 2024Estimating Knowledge in Large Language Models Without Generating a Single Tokencited by 39
  16. 2024Meta-Unlearning on Diffusion Models: Preventing Relearning Unlearned Conceptscited by 39
  17. 2025Exploring Criteria of Loss Reweighting to Enhance LLM Unlearningcited by 39
  18. 2024Digital Forgetting in Large Language Models: A Survey of Unlearning Methodscited by 38
  19. 2023SoK: Memorization in General-Purpose Large Language Modelscited by 36
  20. 2024Selective Forgetting: Advancing Machine Unlearning Techniques and Evaluation in Language Modelscited by 36
  21. 2025Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilitiescited by 36
  22. 2024Mechanistic Unlearning: Robust Knowledge Unlearning and Editing via Mechanistic Localizationcited by 34
  23. 2024Offset Unlearning for Large Language Modelscited by 33
  24. 2024Revisiting Who's Harry Potter: Towards Targeted Unlearning from a Causal Intervention Perspectivecited by 33
  25. 2023Trends in Integration of Knowledge and Large Language Models: A Survey and Taxonomy of Methods, Benchmarks, and Applicationscited by 28
  26. 2024Intrinsic Test of Unlearning Using Parametric Knowledge Tracescited by 28
  27. 2024Gradient Routing: Masking Gradients to Localize Computation in Neural Networkscited by 28
  28. 2024Towards Effective Evaluations and Comparisons for LLM Unlearning Methodscited by 27
  29. 2024On Effects of Steering Latent Representation for Large Language Model Unlearningcited by 27
  30. 2024Towards Scalable Exact Machine Unlearning Using Parameter-Efficient Fine-Tuningcited by 25
  31. 2024Dissecting Fine-Tuning Unlearning in Large Language Modelscited by 25
  32. 2024WAGLE: Strategic Weight Attribution for Effective and Modular Unlearning in Large Language Modelscited by 25
  33. 2024Fundamental Problems With Model Editing: How Should Rational Belief Revision Work in LLMs?cited by 24
  34. 2024Towards Robust Knowledge Unlearning: An Adversarial Framework for Assessing and Improving Unlearning Robustness in Large Language Modelscited by 24
  35. 2025Attractive Metadata Attack: Inducing LLM Agents to Invoke Malicious Toolscited by 23
  36. 2024Cross-Modal Safety Alignment: Is textual unlearning all you need?cited by 22
  37. 2024Learning to Refuse: Towards Mitigating Privacy Risks in LLMscited by 22
  38. 2024A Probabilistic Perspective on Unlearning and Alignment for Large Language Modelscited by 21

50 most cited of 156 All 156 (JSON)