mlsanityData to 13 Oct 2026

Modifying Memories in Transformer Models

arXiv 2012.00363 ↗First posted 1 Dec 2020cs.CL cs.LGcited by 222

Not on the map. Why?

Abstract (arXiv)

Large Transformer models have achieved impressive performance in many natural language tasks. In particular, Transformer based language models have been shown to have great capabilities in encoding factual knowledge in their vast amount of parameters. While the tasks of improving the memorization and generalization of Transformers have been widely studied, it is not well known how to make transformers forget specific old facts and memorize new ones. In this paper, we propose a new task of explicitly modifying specific factual knowledge in Transformer models while ensuring the model performance does not degrade on the unmodified facts. This task is useful in many scenarios, such as updating stale knowledge, protecting privacy, and eliminating unintended biases stored in the models. We benchmarked several approaches that provide natural baseline performances on this task. This leads to the discovery of key components of a Transformer model that are especially effective for knowledge modifications. The work also provides insights into the role that different training phases (such as pretraining and fine-tuning) play towards memorization and knowledge modification.

Key facts. Cited by 222 papers in the map (data through 13 Oct 2026). First posted on arXiv 1 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 7; 2022: 19; 2023: 39; 2024: 78; 2025: 55; 2026: 24.

222 papers.

Sort:

By area:

  1. 2022Locating and Editing Factual Associations in GPTcited by 2,650
  2. 2022Editing Models with Task Arithmeticcited by 1,101
  3. 2022Mass-Editing Memory in a Transformercited by 990
  4. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 841
  5. 2021Editing Factual Knowledge in Language Modelscited by 551
  6. 2021Fast Model Editing at Scalecited by 493
  7. 2022Memory-Based Model Editing at Scalecited by 446
  8. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 404
  9. 2023Editing Large Language Models: Problems, Methods, and Opportunitiescited by 370
  10. 2023MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questionscited by 297
  11. 2022Post-hoc Concept Bottleneck Modelscited by 292
  12. 2021Time-Aware Language Models as Temporal Knowledge Basescited by 291
38 more
  1. 2022Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptorscited by 270
  2. 2023Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Modelscited by 240
  3. 2024AlphaEdit: Null-Space Constrained Knowledge Editing for Language Modelscited by 232
  4. 2024Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?cited by 229
  5. 2021Mind the Gap: Assessing Temporal Generalization in Neural Language Modelscited by 226
  6. 2023Transformer-Patcher: One Mistake worth One Neuroncited by 197
  7. 2023Knowledge Editing for Large Language Models: A Surveycited by 193
  8. 2021Towards Continual Knowledge Learning of Language Modelscited by 173
  9. 2022Large Language Models with Controllable Working Memorycited by 169
  10. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 158
  11. 2023Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attackscited by 157
  12. 2022A Review on Language Models as Knowledge Basescited by 156
  13. 2023Taken out of context: On measuring situational awareness in LLMscited by 137
  14. 2023Editing Implicit Assumptions in Text-to-Image Diffusion Modelscited by 129
  15. 2022DisentQA: Disentangling Parametric and Contextual Knowledge with Counterfactual Question Answeringcited by 94
  16. 2022Calibrating Factual Knowledge in Pretrained Language Modelscited by 92
  17. 2024MEMORYLLM: Towards Self-Updatable Large Language Modelscited by 91
  18. 2023Red teaming ChatGPT via Jailbreaking: Bias, Robustness, Reliability and Toxicitycited by 88
  19. 2021Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefscited by 79
  20. 2023Context-faithful Prompting for Large Language Modelscited by 79
  21. 2024Defending Large Language Models Against Jailbreak Attacks via Layer-specific Editingcited by 79
  22. 2023DEPN: Detecting and Editing Privacy Neurons in Pretrained Language Modelscited by 77
  23. 2025Self-Adapting Language Modelscited by 77
  24. 2025AnyEdit: Edit Any Knowledge Encoded in Language Modelscited by 76
  25. 2023Can Neural Network Memorization Be Localized?cited by 68
  26. 2023Can LMs Learn New Entities from Descriptions? Challenges in Propagating Injected Knowledgecited by 67
  27. 2024SelfIE: Self-Interpretation of Large Language Model Embeddingscited by 65
  28. 2023Parameter Efficient Multi-task Model Fusion with Partial Linearizationcited by 64
  29. 2023Detecting Edit Failures In Large Language Models: An Improved Specificity Benchmarkcited by 62
  30. 2024Synthetic continued pretrainingcited by 58
  31. 2022Entity Cloze By Date: What LMs Know About Unseen Entitiescited by 55
  32. 2024Larimar: Large Language Models with Episodic Memory Controlcited by 53
  33. 2024Context-DPO: Aligning Language Models for Context-Faithfulnesscited by 50
  34. 2023Learning from models beyond fine-tuningcited by 48
  35. 2024Understanding Information Storage and Transfer in Multi-modal Large Language Modelscited by 48
  36. 2023Can Language Models be Instructed to Protect Personal Information?cited by 46
  37. 2024Knowledge Graph Enhanced Large Language Model Editingcited by 39
  38. 2024IRCAN: Mitigating Knowledge Conflicts in LLM Generation via Identifying and Reweighting Context-Aware Neuronscited by 37

50 most cited of 222 All 222 (JSON)