mlsanityData to 7 Oct 2026

LEMoE: Advanced Mixture of Experts Adaptor for Lifelong Model Editing of Large Language Models

arXiv 2406.20030 ↗First posted 28 Jun 2024cs.CLcited by 14

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Large language models (LLMs) require continual knowledge updates to stay abreast of the ever-changing world facts, prompting the formulation of lifelong model editing task. While recent years have witnessed the development of various techniques for single and batch editing, these methods either fail to apply or perform sub-optimally when faced with lifelong editing. In this paper, we introduce LEMoE, an advanced Mixture of Experts (MoE) adaptor for lifelong model editing. We first analyze the factors influencing the effectiveness of conventional MoE adaptor in lifelong editing, including catastrophic forgetting, inconsistent routing and order sensitivity. Based on these insights, we propose a tailored module insertion method to achieve lifelong editing, incorporating a novel KV anchor routing to enhance routing consistency between training and inference stage, along with a concise yet effective clustering-based editing order planning. Experimental results demonstrate the effectiveness of our method in lifelong editing, surpassing previous model editing techniques while maintaining outstanding performance in batch editing task. Our code will be available.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 41 of its 61 references that are in the map. First posted on arXiv 28 Jun 2024.

Builds on

41 of 61 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024MEMoE: Enhancing Model Editing with Mixture of Experts Adaptors29 May

Other references · 40

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  5. 2017Neural Discrete Representation Learningcited by 4,650
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  7. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  8. 2019Language Models as Knowledge Bases?cited by 1,922
  9. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  10. 2018Experience Replay for Continual Learningcited by 1,074
  11. 2022Mass-Editing Memory in a Transformercited by 976
  12. 2022Memory-Based Model Editing at Scalecited by 443
28 more
  1. 2017Zero-Shot Relation Extraction via Reading Comprehensioncited by 425
  2. 2019Online Continual Learning with Maximally Interfered Retrievalcited by 376
  3. 2023Editing Large Language Models: Problems, Methods, and Opportunitiescited by 367
  4. 2023Can We Edit Factual Knowledge by In-Context Learning?cited by 284
  5. 2021Post-Training Quantization for Vision Transformercited by 274
  6. 2022Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptorscited by 267
  7. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  8. 2023Transformer-Patcher: One Mistake worth One Neuroncited by 196
  9. 2023PMET: Precise Model Editing in a Transformercited by 170
  10. 2022Large Language Models with Controllable Working Memorycited by 169
  11. 2023Large Language Model Routing with Benchmark Datasetscited by 166
  12. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 148
  13. 2024Continual Learning for Large Language Models: A Surveycited by 144
  14. 2023Understanding Catastrophic Forgetting in Language Models via Implicit Inferencecited by 138
  15. 2022AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuningcited by 137
  16. 2020Exploring and Predicting Transferability across NLP Taskscited by 126
  17. 2019Scalable and Order-robust Continual Learning with Additive Parameter Decompositioncited by 125
  18. 2023Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuningcited by 106
  19. 2022Calibrating Factual Knowledge in Pretrained Language Modelscited by 92
  20. 2023Scaling Expert Language Models with Unsupervised Domain Discoverycited by 59
  21. 2022Sparsely Activated Mixture-of-Experts are Robust Multi-Task Learnerscited by 51
  22. 2022Fixing Model Bugs with Natural Language Patchescited by 32
  23. 2022The Effect of Task Ordering in Continual Learningcited by 30
  24. 2022On Continual Model Refinement in Out-of-Distribution Data Streamscited by 22
  25. 2022Eliciting and Understanding Cross-Task Skills with Task-Level Mixture-of-Expertscited by 12
  26. 2024Consecutive Batch Model Editing with HooK Layerscited by 9
  27. 2021A Principled Approach to Failure Analysis and Model Repairment: Demonstration in Medical Imagingcited by 5
  28. 2024Scalable Model Editing via Customized Expert Networkscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 5; 2026: 5.

14 papers.

Sort:

By area:

  1. 2024Can Knowledge Editing Really Correct Hallucinations?cited by 28
  2. 2024Can Editing LLMs Inject Harm?cited by 22
  3. 2024Lifelong Knowledge Editing for Vision Language Models with Low-Rank Mixture-of-Expertscited by 14
  4. 2025Mitigating Intra- and Inter-modal Forgetting in Continual Learning of Unified Multimodal Modelscited by 6
  5. 2025MoORE: SVD-based Model MoE-ization for Conflict- and Oblivion-Resistant Multi-Task Adaptationcited by 4
  6. 2026MoEEdit: Efficient and Routing-Stable Knowledge Editing for Mixture-of-Experts LLMscited by 4
  7. 2026Beyond Static Models: An Evolving Framework for Continual Learning in Large Language Models across Training Stagescited by 4
  8. 2025LLaVA-CMoE: Towards Continual Mixture of Experts for Large Vision-Language Modelscited by 2
  9. 2026Efficient Continual Learning in Language Models via Thalamically Routed Cortical Columnscited by 2
  10. 2024UniAdapt: A Universal Adapter for Knowledge Calibrationcited by 1
  11. 2025AUTOLAW: Enhancing Legal Compliance in Large Language Models via Case Law Generation and Jury-Inspired Deliberationcited by 1
  12. 2026Self-Consolidating Language Models: Continual Knowledge Incorporation from Contextcited by 1
2 more
  1. 2025An Information-Theoretic Framework for Robust Large Language Model Editingcited by 0
  2. 2026LadderEdit: Edit-Level Residual Compression for Memory-Efficient Lifelong Editing of LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Generalizable Lifelong Model Editing via Preference Optimization
  2. 2024Lifelong Knowledge Editing for LLMs with Retrieval-Augmented Continuous Prompt Learning
  3. 2026Scalable Knowledge Editing for Mixture-of-Experts LLMs via Tensor-Structured Updates
  4. 2024WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models
  5. 2026More Edits, More Stable: Understanding the Lifelong Normalization in Sequential Model Editing
  6. 2023Knowledge Editing for Large Language Models: A Survey
  7. 2025Benchmarking and Rethinking Knowledge Editing for Large Language Models
  8. 2026The Labyrinth and the Thread: Rethinking Regularizations in Sequential Knowledge Editing for Large Language Models