mlsanityData to 7 Oct 2026

Memory3: Language Modeling with Explicit Memory

arXiv 2407.01178 ↗First posted 1 Jul 2024cs.CL cs.AI cs.LGcited by 39

On the map: Retrieval-augmented generation → Generation and context integration

Abstract (arXiv)

The training and inference of large language models (LLMs) are together a costly process that transports knowledge from raw data to meaningful computation. Inspired by the memory hierarchy of the human brain, we reduce this cost by equipping LLMs with explicit memory, a memory format cheaper than model parameters and text retrieval-augmented generation (RAG). Conceptually, with most of its knowledge externalized to explicit memories, the LLM can enjoy a smaller parameter size, training cost, and inference cost, all proportional to the amount of remaining "abstract knowledge". As a preliminary proof of concept, we train from scratch a 2.4B LLM, which achieves better performance than much larger LLMs as well as RAG models, and maintains higher decoding speed than RAG. The model is named Memory3, since explicit memory is the third form of memory in LLMs after implicit memory (model parameters) and working memory (context key-values). We introduce a memory circuitry theory to support the externalization of knowledge, and present novel techniques including a memory sparsification mechanism that makes storage tractable and a two-stage pretraining scheme that facilitates memory formation.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation. Builds on 80 of its 110 references that are in the map. First posted on arXiv 1 Jul 2024.

Builds on

80 of 110 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Shall We Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Study13 Apr
  2. 2020REALM: Retrieval-Augmented Language Model Pre-Training10 Feb

Other references · 78

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  9. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  10. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  11. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  12. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
38 more
  1. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  2. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  3. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  4. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  5. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  6. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  7. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  8. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  9. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  10. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  11. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  12. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  13. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  14. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  15. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  16. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  17. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,050
  18. 2022Toy Models of Superpositioncited by 928
  19. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  20. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  21. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  22. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  23. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  24. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  25. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  26. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  27. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
  28. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuningcited by 354
  29. 2023HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Modelscited by 329
  30. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 300
  31. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 272
  32. 2022Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Modelcited by 264
  33. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 260
  34. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
  35. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 242
  36. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 235
  37. 2019Depth-Adaptive Transformercited by 227
  38. 2022Memorizing Transformerscited by 219

50 most cited of 78 All 78 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 22; 2026: 11.

39 papers.

Sort:

By area:

  1. 2025Memory in the Age of AI Agentscited by 285
  2. 2025LightMem: Lightweight and Efficient Memory-Augmented Generationcited by 163
  3. 2025MemOS: A Memory OS for AI Systemcited by 129
  4. 2025MemGen: Weaving Generative Latent Memory for Self-Evolving Agentscited by 99
  5. 2025AgentFold: Long-Horizon Web Agents with Proactive Context Managementcited by 95
  6. 2025M+: Extending MemoryLLM with Scalable Long-Term Memorycited by 51
  7. 2026A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agentscited by 40
  8. 2025Rethinking Memory in LLM based Agents: Representations, Operations, and Emerging Topicscited by 36
  9. 2025A Survey on Knowledge-Oriented Retrieval-Augmented Generationcited by 34
  10. 2024How Much Can RAG Help the Reasoning of LLM?cited by 31
  11. 2025MemSearcher: Training LLMs to Reason, Search and Manage Memory via End-to-End Reinforcement Learningcited by 26
  12. 2025Learn to Memorize: Optimizing LLM-based Agents with Adaptive Memory Frameworkcited by 25
27 more
  1. 2025IterResearch: Rethinking Long-Horizon Agents with Interaction Scalingcited by 22
  2. 2025Adaptation of Agentic AI: A Survey of Post-Training, Memory, and Skillscited by 19
  3. 2024On The Planning Abilities of OpenAI's o1 Models: Feasibility, Optimality, and Generalizabilitycited by 18
  4. 2025Sculptor: Empowering LLMs with Cognitive Agency via Active Context Managementcited by 17
  5. 2024Towards LifeSpan Cognitive Systemscited by 13
  6. 2025RARE: Retrieval-Augmented Reasoning Modelingcited by 11
  7. 2026Toward Efficient Agents: Memory, Tool learning, and Planningcited by 11
  8. 2024Improving Factuality with Explicit Working Memorycited by 10
  9. 2025Memory-Augmented Transformers: A Systematic Review from Neuroscience Principles to Enhanced Model Architecturescited by 9
  10. 2025MemWeaver: A Hierarchical Memory from Textual Interactive Behaviors for Personalized Generationcited by 8
  11. 2026MemCast: Memory-Driven Time Series Forecasting with Experience-Conditioned Reasoningcited by 8
  12. 2024CMT: A Memory Compression Method for Continual Knowledge Learning of Large Language Modelscited by 6
  13. 2026MemReranker: Reasoning-Aware Reranking for Agent Memory Retrievalcited by 6
  14. 2026MSA: Memory Sparse Attention for Efficient End-to-End Memory Model Scaling to 100M Tokenscited by 5
  15. 2026Metis: Memory Foundation Modelcited by 5
  16. 2025On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Surveycited by 3
  17. 2025Text2Mem: A Unified Memory Operation Language for Memory Operating Systemcited by 3
  18. 2024Schrodinger's Memory: Large Language Modelscited by 2
  19. 2025The Paradox of Poetic Intent in Back-Translation: Evaluating the Quality of Large Language Models in Chinese Translationcited by 2
  20. 2025Memorization and Knowledge Injection in Gated LLMscited by 2
  21. 2026How LoRA Remembers? A Parametric Memory Law for LLM Finetuningcited by 2
  22. 2026MemSFT: Mitigating Alignment Tax with an External Parametric Memorycited by 2
  23. 2025Scalable Complexity Control Facilitates Reasoning Ability of LLMscited by 1
  24. 2025A Lightweight Framework for Trigger-Guided LoRA-Based Self-Adaptation in LLMscited by 1
  25. 2026Position: Hippocampal Explicit Memory Is the Cornerstone for AGIcited by 0
  26. 2026DYNA : Dynamic Episodic Memory Networks for Augmenting Large Language Models with Temporal Knowledge Graphs in Continuous Learningcited by 0
  27. 2026Towards a Formal Definition of Agent Memory: Basis, Span, Optimality, and the Sequential Memory Problemcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MemOS: An Operating System for Memory-Augmented Generation (MAG) in Large Language Models
  2. 2025Pretraining with hierarchical memories: separating long-tail and common knowledge
  3. 2025MLP Memory: A Retriever-Pretrained Memory for Large Language Models
  4. 2025Memory Decoder: A Pretrained, Plug-and-Play Memory for Large Language Models
  5. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memory
  6. 2026TokenMem: Faithful Knowledge Injection for Frozen LLMs
  7. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories
  8. 2026Memory Decoder at Scale: A Pretrained, Parametric Long-Term Memory