mlsanityData to 14 Oct 2026

PMET: Precise Model Editing in a Transformer

arXiv 2308.08742 ↗First posted 17 Aug 2023cs.CL cs.AI cs.LGcited by 173

Not on the map. Why?

Abstract (arXiv)

Model editing techniques modify a minor proportion of knowledge in Large Language Models (LLMs) at a relatively low cost, which have demonstrated notable success. Existing methods assume Transformer Layer (TL) hidden states are values of key-value memories of the Feed-Forward Network (FFN). They usually optimize the TL hidden states to memorize target knowledge and use it to update the weights of the FFN in LLMs. However, the information flow of TL hidden states comes from three parts: Multi-Head Self-Attention (MHSA), FFN, and residual connections. Existing methods neglect the fact that the TL hidden states contains information not specifically required for FFN. Consequently, the performance of model editing decreases. To achieve more precise model editing, we analyze hidden states of MHSA and FFN, finding that MHSA encodes certain general knowledge extraction patterns. This implies that MHSA weights do not require updating when new knowledge is introduced. Based on above findings, we introduce PMET, which simultaneously optimizes Transformer Component (TC, namely MHSA and FFN) hidden states, while only using the optimized TC hidden states of FFN to precisely update FFN weights. Our experiments demonstrate that PMET exhibits state-of-the-art performance on both the COUNTERFACT and zsRE datasets. Our ablation experiments substantiate the effectiveness of our enhancements, further reinforcing the finding that the MHSA encodes certain general knowledge extraction patterns and indicating its storage of a small amount of factual knowledge. Our code is available at https://github.com/xpq-tech/PMET.

Key facts. Cited by 173 papers in the map (data through 14 Oct 2026). Builds on 26 of its 38 references that are in the map. First posted on arXiv 17 Aug 2023.

Builds on

26 of 38 references.

References · 26

  1. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  2. 2023A Survey of Large Language Modelscited by 2,576
  3. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  4. 2019Language Models as Knowledge Bases?cited by 1,936
  5. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  6. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  7. 2022Mass-Editing Memory in a Transformercited by 1,001
  8. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  9. 2021Editing Factual Knowledge in Language Modelscited by 553
  10. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 532
  11. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 528
  12. 2021Fast Model Editing at Scalecited by 498
14 more
  1. 2022Memory-Based Model Editing at Scalecited by 447
  2. 2017Zero-Shot Relation Extraction via Reading Comprehensioncited by 426
  3. 2023Editing Large Language Models: Problems, Methods, and Opportunitiescited by 372
  4. 2019Revealing the Dark Secrets of BERTcited by 301
  5. 2023Can We Edit Factual Knowledge by In-Context Learning?cited by 285
  6. 2023Evaluating the Ripple Effects of Knowledge Editing in Language Modelscited by 242
  7. 2020Editable Neural Networkscited by 155
  8. 2023Inspecting and Editing Knowledge Representations in Language Modelscited by 141
  9. 2020Self-Attention Attribution: Interpreting Information Interactions Inside Transformercited by 125
  10. 2020Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queriescited by 107
  11. 2021Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Basescited by 107
  12. 2022Knowledge Graph Contrastive Learning Based on Relation-Symmetrical Structurecited by 43
  13. 2022How Much Does Attention Actually Attend? Questioning the Importance of Attention in Pretrained Transformerscited by 21
  14. 2019Knowledge Consistency between Neural Networks and Beyondcited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 13; 2024: 72; 2025: 51; 2026: 37.

173 papers.

Sort:

By area:

  1. 2024Rethinking Machine Unlearning for Large Language Modelscited by 257
  2. 2024AlphaEdit: Null-Space Constrained Knowledge Editing for Language Modelscited by 236
  3. 2023Knowledge Editing for Large Language Models: A Surveycited by 194
  4. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 163
  5. 2024BadEdit: Backdooring large language models by model editingcited by 96
  6. 2024Model Editing at Scale leads to Gradual and Catastrophic Forgettingcited by 93
  7. 2024From Matching to Generation: A Survey on Generative Information Retrievalcited by 89
  8. 2024Detoxifying Large Language Models via Knowledge Editingcited by 87
  9. 2024MyVLM: Personalizing VLMs for User-Specific Queriescited by 58
  10. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 51
  11. 2025LLM-based Agents Suffer from Hallucinations: A Survey of Taxonomy, Methods, and Directionscited by 50
  12. 2024Machine Unlearning in Generative AI: A Surveycited by 49
38 more
  1. 2024From Understanding to Utilization: A Survey on Explainability for Large Language Modelscited by 48
  2. 2024A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicinecited by 45
  3. 2023Can We Edit Multimodal Large Language Models?cited by 44
  4. 2024WilKE: Wise-Layer Knowledge Editor for Lifelong Knowledge Editingcited by 40
  5. 2024Crafting Personalized Agents through Retrieval-Augmented Generation on Editable Memory Graphscited by 38
  6. 2024Locate-then-edit for Multi-hop Factual Recall under Knowledge Editingcited by 37
  7. 2024AKEW: Assessing Knowledge Editing in the Wildcited by 36
  8. 2024Multi-hop Question Answering under Temporal Knowledge Editingcited by 36
  9. 2023A Survey on Knowledge Editing of Neural Networkscited by 34
  10. 2024In-Context Editing: Learning Knowledge from Self-Induced Distributionscited by 34
  11. 2025Reinforced Lifelong Editing for Language Modelscited by 34
  12. 2023How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advancescited by 33
  13. 2023Trends in Integration of Knowledge and Large Language Models: A Survey and Taxonomy of Methods, Benchmarks, and Applicationscited by 28
  14. 2024Can Knowledge Editing Really Correct Hallucinations?cited by 28
  15. 2024Learnable Privacy Neurons Localization in Language Modelscited by 27
  16. 2025The Mirage of Model Editing: Revisiting Evaluation in the Wildcited by 27
  17. 2023DocTER: Evaluating Document-based Knowledge Editingcited by 26
  18. 2024Model Editing by Standard Fine-Tuningcited by 23
  19. 2023Editing Personality for Large Language Modelscited by 22
  20. 2023Attention Lens: A Tool for Mechanistically Interpreting the Attention Head Information Retrieval Mechanismcited by 22
  21. 2024Can Editing LLMs Inject Harm?cited by 22
  22. 2023Memory Injections: Correcting Multi-Hop Reasoning Failures during Inference in Transformer-Based Language Modelscited by 21
  23. 2024Editing Conceptual Knowledge for Large Language Modelscited by 21
  24. 2024LiSeCo: Linear Semantic Control for Language Generationcited by 20
  25. 2023DUnE: Dataset for Unified Editingcited by 19
  26. 2024Everything is Editable: Extend Knowledge Editing to Unstructured Data in Large Language Modelscited by 19
  27. 2024CodeUpdateArena: Benchmarking Knowledge Editing on API Updatescited by 19
  28. 2024Representation Shattering in Transformers: A Synthetic Study with Knowledge Editingcited by 19
  29. 2024Multilingual Knowledge Editing with Language-Agnostic Factual Neuronscited by 18
  30. 2024LLM-Based Multi-Hop Question Answering with Knowledge Graph Integration in Evolving Environmentscited by 18
  31. 2024Should We Really Edit Language Models? On the Evaluation of Edited Language Modelscited by 18
  32. 2024InstructEdit: Instruction-based Knowledge Editing for Large Language Modelscited by 17
  33. 2024Knowledge Editing in Language Models via Adapted Direct Preference Optimizationcited by 17
  34. 2024EVEDIT: Event-based Knowledge Editing with Deductive Editing Boundariescited by 16
  35. 2024Large Language Model Bias Mitigation from the Perspective of Knowledge Editingcited by 16
  36. 2024Commonsense Knowledge Editing Based on Free-Text in LLMscited by 16
  37. 2025Position: Editing Large Language Models Poses Serious Safety Riskscited by 15
  38. 2024Has this Fact been Edited? Detecting Knowledge Edits in Language Modelscited by 14

50 most cited of 173 All 173 (JSON)