mlsanityData to 14 Oct 2026

MeMo: Memory as a Model

arXiv 2605.15156 ↗First posted 14 May 2026cs.CL cs.AI cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) achieve strong performance across a wide range of tasks, but remain frozen after pretraining until subsequent updates. Many real-world applications require timely, domain-specific information, motivating the need for efficient mechanisms to incorporate new knowledge. In this paper, we introduce MeMo (Memory as a Model), a modular framework that encodes new knowledge into a dedicated memory model while keeping the LLM parameters unchanged. Compared to existing methods, MeMo offers several advantages: (a) it captures complex cross-document relationships, (b) it is robust to retrieval noise, (c) it avoids catastrophic forgetting in the LLM, (d) it does not require access to the LLM's weights or output logits, enabling plug-and-play integration with both open and proprietary closed-source LLMs, and (e) its retrieval cost is independent of corpus size at inference time. Our experimental results on three benchmarks, BrowseComp-Plus, NarrativeQA, and MuSiQue, show that MeMo achieves strong performance compared to existing methods across diverse settings.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 72 of its 81 references that are in the map. First posted on arXiv 14 May 2026.

Builds on

72 of 81 references.

References · 72

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2024Qwen2.5 Technical Reportcited by 4,942
  7. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  9. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  10. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  11. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  12. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
38 more
  1. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  2. 2023A Survey of Large Language Modelscited by 2,576
  3. 2016Learning without Forgettingcited by 2,496
  4. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  5. 2024From Local to Global: A Graph RAG Approach to Query-Focused Summarizationcited by 1,453
  6. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  7. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  8. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  9. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  10. 2022Editing Models with Task Arithmeticcited by 1,120
  11. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  12. 2023BloombergGPT: A Large Language Model for Financecited by 939
  13. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 916
  14. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 818
  15. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 800
  16. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
  17. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 672
  18. 2024A Survey on Large Language Models for Code Generationcited by 658
  19. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 639
  20. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 613
  21. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  22. 2017The NarrativeQA Reading Comprehension Challengecited by 597
  23. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 472
  24. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 458
  25. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 407
  26. 2023Learning to Compress Prompts with Gist Tokenscited by 390
  27. 2023Adapting Language Models to Compress Contextscited by 356
  28. 2024Knowledge Conflicts for LLMs: A Surveycited by 283
  29. 2024The Power of Noise: Redefining Retrieval for RAG Systemscited by 281
  30. 2023Physics of Language Models: Part 3.1, Knowledge Storage and Extractioncited by 270
  31. 2024MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queriescited by 257
  32. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 254
  33. 2022RealTime QA: What's the Answer Right Now?cited by 244
  34. 2021Sustainable AI: Environmental Implications, Challenges and Opportunitiescited by 242
  35. 2019ERNIE 2.0: A Continual Pre-training Framework for Language Understandingcited by 231
  36. 2025From RAG to Memory: Non-Parametric Continual Learning for Large Language Modelscited by 231
  37. 2025BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agentcited by 200
  38. 2023In-context Autoencoder for Context Compression in a Large Language Modelcited by 199

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Metis: Memory Foundation Modelcited by 7
  2. 2026AutoMem: Automated Learning of Memory as a Cognitive Skillcited by 5
  3. 2026PertMind: Eliciting Emergent Biological Reasoning in LLM via Reinforcement Learning on Cellular Perturbation Datacited by 0
  4. 2026Graph-Based Personalized Memory for LLM Agents: Representation, Evolution, Retrieval, and Evaluationcited by 0