mlsanityData to 14 Oct 2026

Learnable Privacy Neurons Localization in Language Models

arXiv 2405.10989 ↗First posted 16 May 2024cs.LG cs.AI cs.CLcited by 27

Not on the map. Why?

Abstract (arXiv)

Concerns regarding Large Language Models (LLMs) to memorize and disclose private information, particularly Personally Identifiable Information (PII), become prominent within the community. Many efforts have been made to mitigate the privacy risks. However, the mechanism through which LLMs memorize PII remains poorly understood. To bridge this gap, we introduce a pioneering method for pinpointing PII-sensitive neurons (privacy neurons) within LLMs. Our method employs learnable binary weight masks to localize specific neurons that account for the memorization of PII in LLMs through adversarial training. Our investigations discover that PII is memorized by a small subset of neurons across all layers, which shows the property of PII specificity. Furthermore, we propose to validate the potential in PII risk mitigation by deactivating the localized privacy neurons. Both quantitative and qualitative experiments demonstrate the effectiveness of our neuron localization algorithm.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Builds on 39 of its 57 references that are in the map. First posted on arXiv 16 May 2024.

Builds on

39 of 57 references.

References · 39

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  3. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  4. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  5. 2020Extracting Training Data from Large Language Modelscited by 2,189
  6. 2019Language Models as Knowledge Bases?cited by 1,936
  7. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,148
  8. 2022Mass-Editing Memory in a Transformercited by 1,001
  9. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  10. 2021Knowledge Neurons in Pretrained Transformerscited by 601
  11. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 583
  12. 2021Editing Factual Knowledge in Language Modelscited by 553
27 more
  1. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 518
  2. 2022Knowledge Unlearning for Mitigating Privacy Risks in Language Modelscited by 437
  3. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 412
  4. 2021Differentially Private Fine-tuning of Language Modelscited by 400
  5. 2023Who's Harry Potter? Approximate Unlearning in LLMscited by 381
  6. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 314
  7. 2023Analyzing Leakage of Personally Identifiable Information in Language Modelscited by 285
  8. 2023Unlearn What You Want to Forget: Efficient Unlearning for LLMscited by 215
  9. 2019Neural Legal Judgment Prediction in Englishcited by 209
  10. 2022Are Large Pre-Trained Language Models Leaking Your Personal Information?cited by 200
  11. 2020How Context Affects Language Models' Factual Predictionscited by 181
  12. 2021Towards Continual Knowledge Learning of Language Modelscited by 174
  13. 2023PMET: Precise Model Editing in a Transformercited by 173
  14. 2021Deep Learning Through the Lens of Example Difficultycited by 141
  15. 2021Large-Scale Differentially Private BERTcited by 118
  16. 2022Measuring Forgetting of Memorized Training Examplescited by 118
  17. 2020Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queriescited by 107
  18. 2021Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Basescited by 107
  19. 2016De-identification of Patient Notes with Recurrent Neural Networkscited by 88
  20. 2021Large Scale Private Learning via Low-rank Reparametrizationcited by 88
  21. 2023Fast Model Debias with Machine Unlearningcited by 72
  22. 2023Can Neural Network Memorization Be Localized?cited by 69
  23. 2022Exploring the Limits of Differentially Private Deep Learning with Group-wise Clippingcited by 52
  24. 2020FLERT: Document-Level Features for Named Entity Recognitioncited by 45
  25. 2023Identifying and Mitigating Privacy Risks Stemming from Language Models: A Surveycited by 30
  26. 2024Large Language Model Bias Mitigation from the Perspective of Knowledge Editingcited by 16
  27. 2020Sensitive Data Detection and Classification in Spanish Clinical Text: Experiments with BERTcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 9; 2025: 15; 2026: 3.

27 papers.

Sort:

By area:

  1. 2024PAD: Personalized Alignment of LLMs at Decoding-Timecited by 60
  2. 2024Large Language Model Safety: A Holistic Surveycited by 59
  3. 2024On Large Language Model Continual Unlearningcited by 52
  4. 2025FairSteer: Inference Time Debiasing for LLMs with Dynamic Activation Steeringcited by 40
  5. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 27
  6. 2024Towards Understanding Multi-Task Learning (Generalization) of LLMs via Detecting and Exploring Task-Specific Neuronscited by 25
  7. 2025Datasets and Recipes for Video Temporal Grounding via Reinforcement Learningcited by 19
  8. 2024Identifying Query-Relevant Neurons in Large Language Models for Long-Form Textscited by 14
  9. 2025Persona-judge: Personalized Alignment of Large Language Models via Token-level Self-judgmentcited by 11
  10. 2024KIF: Knowledge Identification and Fusion for Language Model Continual Learningcited by 9
  11. 2025Memorization Sinks: Isolating Memorization during LLM Trainingcited by 7
  12. 2025The Achilles' Heel of LLMs: How Altering a Handful of Neurons Can Cripple Language Abilitiescited by 7
15 more
  1. 2025PrivacyScalpel: Enhancing LLM Privacy via Interpretable Feature Intervention with Sparse Autoencoderscited by 6
  2. 2025BiasFilter: An Inference-Time Debiasing Framework for Large Language Modelscited by 6
  3. 2025Understanding and Mitigating Political Stance Cross-topic Generalization in Large Language Modelscited by 6
  4. 2024Property Neurons in Self-Supervised Speech Transformerscited by 5
  5. 2024The Tug of War Within: Mitigating the Fairness-Privacy Conflicts in Large Language Modelscited by 5
  6. 2025DiffPO: Diffusion-styled Preference Optimization for Efficient Inference-Time Alignment of Large Language Modelscited by 5
  7. 2025BiasGuard: A Reasoning-enhanced Bias Detection Tool For Large Language Modelscited by 4
  8. 2026Why Does Reinforcement Learning Generalize? A Feature-Level Mechanistic Study of Post-Training in Large Language Modelscited by 3
  9. 2025PATCH: Mitigating PII Leakage in Language Models with Privacy-Aware Targeted Circuit PatcHingcited by 2
  10. 2025Detection, Classification, and Mitigation of Gender Bias in Large Language Modelscited by 1
  11. 2025The Model's Language Matters: A Comparative Privacy Analysis of LLMscited by 1
  12. 2025Leverage Unlearning to Sanitize LLMscited by 1
  13. 2026Discovering Universal Activation Directions for PII Leakage in Language Modelscited by 1
  14. 2024Learning from Litigation: Graphs and LLMs for Retrieval and Reasoning in eDiscoverycited by 0
  15. 2025Towards the Anonymization of the Language Modelingcited by 0