mlsanityData to 7 Oct 2026

Do Localization Methods Actually Localize Memorized Data in LLMs? A Tale of Two Benchmarks

arXiv 2311.09060 ↗First posted 15 Nov 2023cs.CLcited by 29

Not on the map. Why?

Abstract (arXiv)

The concept of localization in LLMs is often mentioned in prior work; however, methods for localization have never been systematically and directly evaluated. We propose two complementary benchmarks that evaluate the ability of localization methods to pinpoint LLM components responsible for memorized data. In our INJ benchmark, we actively inject a piece of new information into a small subset of LLM weights, enabling us to directly evaluate whether localization methods can identify these "ground truth" weights. In our DEL benchmark, we evaluate localization by measuring how much dropping out identified neurons deletes a memorized pretrained sequence. Despite their different perspectives, our two benchmarks yield consistent rankings of five localization methods. Methods adapted from network pruning perform well on both benchmarks, and all evaluated methods show promising localization ability. On the other hand, even successful methods identify neurons that are not specific to a single memorized sequence.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Builds on 40 of its 56 references that are in the map. First posted on arXiv 15 Nov 2023.

Builds on

40 of 56 references.

References · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  3. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  4. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  5. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  6. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  7. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  8. 2020Extracting Training Data from Large Language Modelscited by 2,144
  9. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  10. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  11. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
  12. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
28 more
  1. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  2. 2019Machine Unlearningcited by 1,020
  3. 2022In-context Learning and Induction Headscited by 930
  4. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  5. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  6. 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 673
  7. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 571
  8. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  9. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  10. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  11. 2023Evaluating the Ripple Effects of Knowledge Editing in Language Modelscited by 238
  12. 2023Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Modelscited by 237
  13. 2016Understanding Neural Networks through Representation Erasurecited by 223
  14. 2023Localizing Model Behavior with Path Patchingcited by 182
  15. 2021Counterfactual Memorization in Neural Language Modelscited by 155
  16. 2020Editable Neural Networkscited by 154
  17. 2017Learning to Generate Reviews and Discovering Sentimentcited by 147
  18. 2023Task-Specific Skill Localization in Fine-tuned Language Modelscited by 104
  19. 2021Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?cited by 91
  20. 2020EarlyBERT: Efficient BERT Training via Early-bird Lottery Ticketscited by 79
  21. 2023Can LMs Learn New Entities from Descriptions? Challenges in Propagating Injected Knowledgecited by 66
  22. 2023Can Neural Network Memorization Be Localized?cited by 66
  23. 2021Low-Complexity Probing via Finding Subnetworkscited by 63
  24. 2023Detecting Edit Failures In Large Language Models: An Improved Specificity Benchmarkcited by 62
  25. 2022Entity Cloze By Date: What LMs Know About Unseen Entitiescited by 55
  26. 2020Understanding Unintended Memorization in Federated Learningcited by 36
  27. 2022Discovering Language-neutral Sub-networks in Multilingual Language Modelscited by 28
  28. 2021Do Question Answering Modeling Improvements Hold Across Benchmarks?cited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 10; 2026: 9.

29 papers.

Sort:

By area:

  1. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 157
  2. 2024Position: LLM Unlearning Benchmarks are Weak Measures of Progresscited by 58
  3. 2024On Large Language Model Continual Unlearningcited by 51
  4. 2024Localizing Paragraph Memorization in Language Modelscited by 30
  5. 2024Intrinsic Test of Unlearning Using Parametric Knowledge Tracescited by 28
  6. 2024Mitigating Memorization In Language Modelscited by 11
  7. 2025Does Localization Inform Unlearning? A Rigorous Examination of Local Parameter Attribution for Knowledge Unlearning in Language Modelscited by 10
  8. 2024Localizing Memorization in SSL Vision Encoderscited by 9
  9. 2025CaKE: Circuit-aware Editing Enables Generalizable Knowledge Learnerscited by 9
  10. 2025The Landscape of Memorization in LLMs: Mechanisms, Measurement, and Mitigationcited by 9
  11. 2025Mitigating Memorization in LLMs using Activation Steeringcited by 7
  12. 2025Memorization Sinks: Isolating Memorization during LLM Trainingcited by 7
17 more
  1. 2024When Parts Are Greater Than Sums: Individual LLM Components Can Outperform Full Modelscited by 6
  2. 2025Private Memorization Editing: Turning Memorization into a Defense to Strengthen Data Privacy in Large Language Modelscited by 6
  3. 2026Knowledge Localization in Mixture-of-Experts LLMs Using Cross-Lingual Inconsistencycited by 6
  4. 2026Benchmarking Empirical Privacy Protection for Adaptations of Large Language Modelscited by 4
  5. 2024Detecting Memorization in Large Language Modelscited by 3
  6. 2024Large Language Models as Carriers of Hidden Messagescited by 2
  7. 2025Breaking Down Bias: On The Limits of Generalizable Pruning Strategiescited by 2
  8. 2026GSS: Gated Subspace Steering for Selective Memorization Mitigation in LLMscited by 2
  9. 2026LACUNA: A Testbed for Evaluating Localization Precision for LLM Unlearningcited by 2
  10. 2025A Lightweight Method to Disrupt Memorized Sequences in LLMcited by 1
  11. 2025From Memorization to Reasoning in the Spectrum of Loss Curvaturecited by 1
  12. 2026Data-Free Privacy-Preserving for LLMs via Model Inversion and Selective Unlearningcited by 1
  13. 2026Natively Unlearnable Large Language Modelscited by 1
  14. 2025Beyond Frequency: The Role of Redundancy in Large Language Model Memorizationcited by 0
  15. 2026Output Vector Editing for Memorization Mitigation in Large Language Modelscited by 0
  16. 2026Don't Forget! Decomposing the Training Dynamics of Memorization in Language Modelscited by 0
  17. 2026Weights Read and Write Features: Scalable Parameter Decomposition Grounded in Activation Spacecited by 0