mlsanityData to 7 Oct 2026

DEPN: Detecting and Editing Privacy Neurons in Pretrained Language Models

arXiv 2310.20138 ↗First posted 31 Oct 2023cs.CR cs.CLcited by 75

Not on the map. Why?

Cited by 53 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models pretrained on a huge amount of data capture rich knowledge and information in the training data. The ability of data memorization and regurgitation in pretrained language models, revealed in previous studies, brings the risk of data leakage. In order to effectively reduce these risks, we propose a framework DEPN to Detect and Edit Privacy Neurons in pretrained language models, partially inspired by knowledge neurons and model editing. In DEPN, we introduce a novel method, termed as privacy neuron detector, to locate neurons associated with private information, and then edit these detected privacy neurons by setting their activations to zero. Furthermore, we propose a privacy neuron aggregator dememorize private information in a batch processing manner. Experimental results show that our method can significantly and efficiently reduce the exposure of private data leakage without deteriorating the performance of the model. Additionally, we empirically demonstrate the relationship between model memorization and privacy neurons, from multiple perspectives, including model size, training time, prompts, privacy neuron distribution, illustrating the robustness of our approach.

Key facts. Cited by 75 papers in the map (data through 7 Oct 2026). Builds on 22 of its 31 references that are in the map. First posted on arXiv 31 Oct 2023.

Builds on

22 of 31 references.

References · 22

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  5. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  6. 2020Extracting Training Data from Large Language Modelscited by 2,144
  7. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  8. 2019Machine Unlearningcited by 1,020
  9. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  10. 2021Editing Factual Knowledge in Language Modelscited by 548
  11. 2022Memory-Based Model Editing at Scalecited by 443
  12. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
10 more
  1. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 320
  2. 2020Descent-to-Delete: Gradient-Based Methods for Machine Unlearningcited by 303
  3. 2020Modifying Memories in Transformer Modelscited by 222
  4. 2022What Does it Mean for a Language Model to Preserve Privacy?cited by 194
  5. 2021Adaptive Machine Unlearningcited by 164
  6. 2017Ethical Challenges in Data-Driven Dialogue Systemscited by 86
  7. 2021Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristicscited by 67
  8. 2018Privacy-preserving Neural Representations of Textcited by 54
  9. 2021CAPE: Context-Aware Private Embeddings for Private Language Learningcited by 22
  10. 2020Sensitive Data Detection and Classification in Spanish Clinical Text: Experiments with BERTcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 35; 2025: 31; 2026: 9.

75 papers.

Sort:

By area:

  1. 2024MUSE: Machine Unlearning Six-Way Evaluation for Language Modelscited by 251
  2. 2024Rethinking Machine Unlearning for Large Language Modelscited by 247
  3. 2024Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearningcited by 160
  4. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 158
  5. 2024Eight Methods to Evaluate Robust Unlearning in LLMscited by 153
  6. 2024Tamper-Resistant Safeguards for Open-Weight LLMscited by 132
  7. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 123
  8. 2024An Adversarial Perspective on Machine Unlearning for AI Safetycited by 113
  9. 2024Large Language Model Unlearning via Embedding-Corrupted Promptscited by 112
  10. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  11. 2024BadEdit: Backdooring large language models by model editingcited by 84
  12. 2024Defending Large Language Models Against Jailbreak Attacks via Layer-specific Editingcited by 77
38 more
  1. 2024Catastrophic Failure of LLM Unlearning via Quantizationcited by 76
  2. 2025Towards LLM Unlearning Resilient to Relearning Attacks: A Sharpness-Aware Minimization Perspective and Beyondcited by 76
  3. 2024LLM Unlearning via Loss Adjustment with Only Forget Datacited by 71
  4. 2024Understanding LLMs: A Comprehensive Overview from Training to Inferencecited by 58
  5. 2024On Large Language Model Continual Unlearningcited by 51
  6. 2024Challenging Forgets: Unveiling the Worst-Case Forget Sets in Machine Unlearningcited by 44
  7. 2024Evaluating Copyright Takedown Methods for Language Modelscited by 42
  8. 2024Digital Forgetting in Large Language Models: A Survey of Unlearning Methodscited by 38
  9. 2024IRCAN: Mitigating Knowledge Conflicts in LLM Generation via Identifying and Reweighting Context-Aware Neuronscited by 36
  10. 2024Towards Robust and Parameter-Efficient Knowledge Unlearning for LLMscited by 36
  11. 2024Offset Unlearning for Large Language Modelscited by 33
  12. 2024In-Context Editing: Learning Knowledge from Self-Induced Distributionscited by 33
  13. 2025A Comprehensive Survey of Machine Unlearning Techniques for Large Language Modelscited by 30
  14. 2024WAGLE: Strategic Weight Attribution for Effective and Modular Unlearning in Large Language Modelscited by 25
  15. 2024Fundamental Problems With Model Editing: How Should Rational Belief Revision Work in LLMs?cited by 24
  16. 2024Extracting Unlearned Information from LLMs with Activation Steeringcited by 19
  17. 2025RULE: Reinforcement UnLEarning Achieves Forget-Retain Pareto Optimalitycited by 14
  18. 2024REVS: Unlearning Sensitive Information in Language Models via Rank Editing in the Vocabulary Spacecited by 12
  19. 2025Reasoning Model Unlearning: Forgetting Traces, Not Just Answers, While Preserving Reasoning Skillscited by 12
  20. 2024Mitigating Memorization In Language Modelscited by 11
  21. 2025LLM Unlearning Reveals a Stronger-Than-Expected Coreset Effect in Current Benchmarkscited by 11
  22. 2024Editing Factual Knowledge and Explanatory Ability of Medical Large Language Modelscited by 10
  23. 2025Unlearning Isn't Invisible: Detecting Unlearning Traces in LLMs from Model Outputscited by 10
  24. 2025Proactive Privacy Amnesia for Large Language Models: Safeguarding PII with Negligible Impact on Model Utilitycited by 9
  25. 2025Unlearning That Lasts: Utility-Preserving, Robust, and Almost Irreversible Forgetting in LLMscited by 9
  26. 2025MLLMEraser: Achieving Test-Time Unlearning in Multimodal Large Language Models through Activation Steeringcited by 9
  27. 2024An Exploratory Investigation into Code License Infringements in Large Language Model Training Datasetscited by 8
  28. 2024U Can't Gen This? A Survey of Intellectual Property Protection Methods for Data in Generative AIcited by 7
  29. 2025CLUE: Conflict-guided Localization for LLM Unlearning Frameworkcited by 7
  30. 2025DRAGON: Guard LLM Unlearning in Context via Negative Detection and Reasoningcited by 7
  31. 2024PII-Scope: A Comprehensive Study on Training Data PII Extraction Attacks in LLMscited by 6
  32. 2025ParaPO: Aligning Language Models to Reduce Verbatim Reproduction of Pre-training Datacited by 6
  33. 2025Step-by-Step Reasoning Attack: Revealing 'Erased' Knowledge in Large Language Modelscited by 6
  34. 2025Erasing Without Remembering: Implicit Knowledge Forgetting in Large Language Modelscited by 5
  35. 2026Is Gradient Ascent Really Necessary? Memorize to Forget for Machine Unlearningcited by 5
  36. 2024Get Confused Cautiously: Textual Sequence Memorization Erasure with Selective Entropy Maximizationcited by 4
  37. 2024Learning Where to Edit Vision Transformerscited by 4
  38. 2025SUV: Scalable Large Language Model Copyright Compliance with Regularized Selective Unlearningcited by 4

50 most cited of 75 All 75 (JSON)