mlsanityData to 7 Oct 2026

Memorization and Knowledge Injection in Gated LLMs

arXiv 2504.21239 ↗First posted 30 Apr 2025cs.CL cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) currently struggle to sequentially add new memories and integrate new knowledge. These limitations contrast with the human ability to continuously learn from new experiences and acquire knowledge throughout life. Most existing approaches add memories either through large context windows or external memory buffers (e.g., Retrieval-Augmented Generation), and studies on knowledge injection rarely test scenarios resembling everyday life events. In this work, we introduce a continual learning framework, Memory Embedded in Gated LLMs (MEGa), which injects event memories directly into the weights of LLMs. Each memory is stored in a dedicated set of gated low-rank weights. During inference, a gating mechanism activates relevant memory weights by matching query embeddings to stored memory embeddings. This enables the model to both recall entire memories and answer related questions. On two datasets - fictional characters and Wikipedia events - MEGa outperforms baseline approaches in mitigating catastrophic forgetting. Our model draws inspiration from the complementary memory system of the human brain.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 72 of its 119 references that are in the map. First posted on arXiv 30 Apr 2025.

Builds on

72 of 119 references.

References · 72

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  6. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  7. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  8. 2024Mixtral of Expertscited by 1,785
  9. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  10. 2022Mass-Editing Memory in a Transformercited by 976
  11. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
  12. 2021Fast Model Editing at Scalecited by 491
38 more
  1. 2017Zero-Shot Relation Extraction via Reading Comprehensioncited by 425
  2. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 399
  3. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 341
  4. 2024LoRA Learns Less and Forgets Lesscited by 313
  5. 2024HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Modelscited by 273
  6. 2022Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptorscited by 267
  7. 2023How is ChatGPT's behavior changing over time?cited by 265
  8. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  9. 2020Revisiting Few-sample BERT Fine-tuningcited by 237
  10. 2023A Rank Stabilization Scaling Factor for Fine-Tuning with LoRAcited by 231
  11. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  12. 2024Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?cited by 227
  13. 2022SGPT: GPT Sentence Embeddings for Semantic Searchcited by 204
  14. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 192
  15. 2023Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMscited by 175
  16. 2024Retrieval-Augmented Generation with Graphs (GraphRAG)cited by 170
  17. 2020Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgettingcited by 169
  18. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 157
  19. 2019LAMOL: LAnguage MOdeling for Lifelong Language Learningcited by 156
  20. 2019Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Modelscited by 141
  21. 2023Understanding Catastrophic Forgetting in Language Models via Implicit Inferencecited by 138
  22. 2023Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluationcited by 134
  23. 2024MemoRAG: Boosting Long Context Processing with Global Memory-Enhanced Retrieval Augmentationcited by 118
  24. 2024Mitigating Catastrophic Forgetting in Large Language Models with Self-Synthesized Rehearsalcited by 114
  25. 2024Model Editing at Scale leads to Gradual and Catastrophic Forgettingcited by 92
  26. 2024Unfamiliar Finetuning Examples Control How Language Models Hallucinatecited by 85
  27. 2023MELO: Enhancing Model Editing with Neuron-Indexed Dynamic LoRAcited by 80
  28. 2024LoraRetriever: Input-Aware LoRA Retrieval and Composition for Mixed Tasks in the Wildcited by 68
  29. 2023Turning large language models into cognitive modelscited by 66
  30. 2024Human-inspired Episodic Memory for Infinite Context LLMscited by 62
  31. 2024CogBench: a large language model walks into a psychology labcited by 60
  32. 2024Injecting New Knowledge into Large Language Models via Supervised Fine-Tuningcited by 56
  33. 2024The Butterfly Effect of Model Editing: Few Edits Can Trigger Large Language Models Collapsecited by 54
  34. 2024Model Tailor: Mitigating Catastrophic Forgetting in Multi-modal Large Language Modelscited by 54
  35. 2024Larimar: Large Language Models with Episodic Memory Controlcited by 53
  36. 2024Memory3: Language Modeling with Explicit Memorycited by 39
  37. 2022The Neural Race Reduction: Dynamics of Abstraction in Gated Networkscited by 37
  38. 2024Unveiling the Generalization Power of Fine-Tuned Large Language Modelscited by 35

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025A Survey of Context Engineering for Large Language Modelscited by 120
  2. 2025Diffusion-Inspired Masked Fine-Tuning for Knowledge Injection in Autoregressive LLMscited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2025Catastrophic Forgetting in LLMs: A Comparative Analysis Across Language Tasks
  2. 2025Comparing Knowledge Injection Methods for LLMs in a Low-Resource Regime
  3. 2026TokenMem: Faithful Knowledge Injection for Frozen LLMs
  4. 2023Think-in-Memory: Recalling and Post-thinking Enable LLMs with Long-Term Memory
  5. 2025Memory in Large Language Models: Mechanisms, Evaluation and Evolution
  6. 2026Is One Score Enough? Rethinking the Evaluation of Sequentially Evolving LLM Memory
  7. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories
  8. 2026Human-Inspired Memory Architecture for LLM Agents