# WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models

arXiv: https://arxiv.org/abs/2405.14768 · HTML: https://mlsanity.com/paper/2405.14768/

## Key facts

- Cited by 92 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning.
- Builds on 15 of its 111 references that are in the map.
- First posted on arXiv 23 May 2024.

## Placed in

- LLM reasoning / [Knowledge- and tool-augmented reasoning](https://mlsanity.com/lr/knowledge-and-tool-augmented-reasoning/)

## Builds on (most cited first, 15 of 15)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [MemGPT: Towards LLMs as Operating Systems](https://mlsanity.com/paper/2310.08560/) (2023)
- [Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval](https://mlsanity.com/paper/2007.00808/) (2020)
- [An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning](https://mlsanity.com/paper/2308.08747/) (2023)
- [ShortGPT: Layers in Large Language Models are More Redundant Than You Expect](https://mlsanity.com/paper/2403.03853/) (2024)
- [Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention](https://mlsanity.com/paper/2404.07143/) (2024)
- [Arcee's MergeKit: A Toolkit for Merging Large Language Models](https://mlsanity.com/paper/2403.13257/) (2024)
- [Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs](https://mlsanity.com/paper/2312.05934/) (2023)
- [A Comprehensive Study of Knowledge Editing for Large Language Models](https://mlsanity.com/paper/2401.01286/) (2024)
- [MEMORYLLM: Towards Self-Updatable Large Language Models](https://mlsanity.com/paper/2402.04624/) (2024)
- [Deep Model Fusion: A Survey](https://mlsanity.com/paper/2309.15698/) (2023)
- [Memoria: Resolving Fateful Forgetting Problem through Human-Inspired Memory Architecture](https://mlsanity.com/paper/2310.03052/) (2023)
- [Improving Model Fusion by Training-time Neuron Alignment with Fixed Neuron Anchors](https://mlsanity.com/paper/2402.01342/) (2024)

## Built on by (most cited first, 30 of 92)

- [AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models](https://mlsanity.com/paper/2410.02355/) (2024)
- [A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deployment](https://mlsanity.com/paper/2504.15585/) (2025)
- [MemGen: Weaving Generative Latent Memory for Self-Evolving Agents](https://mlsanity.com/paper/2509.24704/) (2025)
- [Lifelong Learning of Large Language Model based Agents: A Roadmap](https://mlsanity.com/paper/2501.07278/) (2025)
- [VisMem: Latent Vision Memory Unlocks Potential of Vision-Language Models](https://mlsanity.com/paper/2511.11007/) (2025)
- [Position: Episodic Memory is the Missing Piece for Long-Term LLM Agents](https://mlsanity.com/paper/2502.06975/) (2025)
- [In-Context Editing: Learning Knowledge from Self-Induced Distributions](https://mlsanity.com/paper/2406.11194/) (2024)
- [Can Knowledge Editing Really Correct Hallucinations?](https://mlsanity.com/paper/2410.16251/) (2024)
- [A Practitioner's Guide to Continual Multimodal Pretraining](https://mlsanity.com/paper/2408.14471/) (2024)
- [Can Editing LLMs Inject Harm?](https://mlsanity.com/paper/2407.20224/) (2024)
- [Continual Learning via Sparse Memory Finetuning](https://mlsanity.com/paper/2510.15103/) (2025)
- [Unlocking Efficient, Scalable, and Continual Knowledge Editing with Basis-Level Representation Fine-Tuning](https://mlsanity.com/paper/2503.00306/) (2025)
- [Towards LifeSpan Cognitive Systems](https://mlsanity.com/paper/2409.13265/) (2024)
- [LiT: Delving into a Simple Linear Diffusion Transformer for Image Generation](https://mlsanity.com/paper/2501.12976/) (2025)
- [MEMOIR: Lifelong Model Editing with Minimal Overwrite and Informed Retention for LLMs](https://mlsanity.com/paper/2506.07899/) (2025)
- [UltraEdit: Training-, Subject-, and Memory-Free Lifelong Editing in Language Models](https://mlsanity.com/paper/2505.14679/) (2025)
- [Fine-tuning Done Right in Model Editing](https://mlsanity.com/paper/2509.22072/) (2025)
- [ChroKnowledge: Unveiling Chronological Knowledge of Language Models in Multiple Domains](https://mlsanity.com/paper/2410.09870/) (2024)
- [O-Edit: Orthogonal Subspace Editing for Language Model Sequential Editing](https://mlsanity.com/paper/2410.11469/) (2024)
- [Revealing and Mitigating Over-Attention in Knowledge Editing](https://mlsanity.com/paper/2502.14838/) (2025)
- [Knowledge Fusion of Large Language Models Via Modular SkillPacks](https://mlsanity.com/paper/2505.18502/) (2025)
- [Locas: Your Models are Principled Initializers of Locally-Supported Parametric Memories](https://mlsanity.com/paper/2602.05085/) (2026)
- [Identifying Knowledge Editing Types in Large Language Models](https://mlsanity.com/paper/2409.19663/) (2024)
- [Tracing and Reversing Edits in LLMs](https://mlsanity.com/paper/2505.20819/) (2025)
- [Is Model Editing Built on Sand? Revealing Its Illusory Success and Fragile Foundation](https://mlsanity.com/paper/2510.00625/) (2025)
- [Edit Less, Achieve More: Dynamic Sparse Neuron Masking for Lifelong Knowledge Editing in LLMs](https://mlsanity.com/paper/2510.22139/) (2025)
- [μKE: Matryoshka Unstructured Knowledge Editing of Large Language Models](https://mlsanity.com/paper/2504.01196/) (2025)
- [MedREK: Retrieval-Based Editing for Medical LLMs with Key-Aware Prompts](https://mlsanity.com/paper/2510.13500/) (2025)
- [CrispEdit: Low-Curvature Projections for Scalable Non-Destructive LLM Editing](https://mlsanity.com/paper/2602.15823/) (2026)
- [LoRASuite: Efficient LoRA Adaptation Across Large Language Model Upgrades](https://mlsanity.com/paper/2505.13515/) (2025)

## Abstract (arXiv)

Large language models (LLMs) need knowledge updates to meet the ever-growing world facts and correct the hallucinated responses, facilitating the methods of lifelong model editing. Where the updated knowledge resides in memories is a fundamental question for model editing. In this paper, we find that editing either long-term memory (direct model parameters) or working memory (non-parametric knowledge of neural network activations/representations by retrieval) will result in an impossible triangle -- reliability, generalization, and locality can not be realized together in the lifelong editing settings. For long-term memory, directly editing the parameters will cause conflicts with irrelevant pretrained knowledge or previous edits (poor reliability and locality). For working memory, retrieval-based activations can hardly make the model understand the edits and generalize (poor generalization). Therefore, we propose WISE to bridge the gap between memories. In WISE, we design a dual parametric memory scheme, which consists of the main memory for the pretrained knowledge and a side memory for the edited knowledge. We only edit the knowledge in the side memory and train a router to decide which memory to go through when given a query. For continual editing, we devise a knowledge-sharding mechanism where different sets of edits reside in distinct subspaces of parameters, and are subsequently merged into a shared memory without conflicts. Extensive experiments show that WISE can outperform previous model editing methods and overcome the impossible triangle under lifelong model editing of question answering, hallucination, and out-of-distribution settings across trending LLM architectures, e.g., GPT, LLaMA, and Mistral. Code is available at https://github.com/zjunlp/EasyEdit.