# Larimar: Large Language Models with Episodic Memory Control

arXiv: https://arxiv.org/abs/2403.11901 · HTML: https://mlsanity.com/paper/2403.11901/

## Key facts

- Cited by 53 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 60 references that are in the map.
- First posted on arXiv 18 Mar 2024.

## Builds on (most cited first, 30 of 41)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Mass-Editing Memory in a Transformer](https://mlsanity.com/paper/2210.07229/) (2022)
- [Generalization through Memorization: Nearest Neighbor Language Models](https://mlsanity.com/paper/1911.00172/) (2019)
- [Neural Turing Machines](https://mlsanity.com/paper/1410.5401/) (2014)
- [Knowledge Neurons in Pretrained Transformers](https://mlsanity.com/paper/2104.08696/) (2021)
- [Editing Factual Knowledge in Language Models](https://mlsanity.com/paper/2104.08164/) (2021)
- [Fast Model Editing at Scale](https://mlsanity.com/paper/2110.11309/) (2021)
- [Memory-Based Model Editing at Scale](https://mlsanity.com/paper/2206.06520/) (2022)
- [Zero-Shot Relation Extraction via Reading Comprehension](https://mlsanity.com/paper/1706.04115/) (2017)
- [Editing Large Language Models: Problems, Methods, and Opportunities](https://mlsanity.com/paper/2305.13172/) (2023)
- [Memory Networks](https://mlsanity.com/paper/1410.3916/) (2014)
- [The Impact of Positional Encoding on Length Generalization in Transformers](https://mlsanity.com/paper/2305.19466/) (2023)
- [Can We Edit Factual Knowledge by In-Context Learning?](https://mlsanity.com/paper/2305.12740/) (2023)
- [Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors](https://mlsanity.com/paper/2211.11031/) (2022)
- [Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models](https://mlsanity.com/paper/2301.04213/) (2023)
- [Modifying Memories in Transformer Models](https://mlsanity.com/paper/2012.00363/) (2020)
- [In-Context Unlearning: Language Models as Few Shot Unlearners](https://mlsanity.com/paper/2310.07579/) (2023)
- [Key-Value Memory Networks for Directly Reading Documents](https://mlsanity.com/paper/1606.03126/) (2016)
- [Repeat After Me: Transformers are Better than State Space Models at Copying](https://mlsanity.com/paper/2402.01032/) (2024)
- [Unlearn What You Want to Forget: Efficient Unlearning for LLMs](https://mlsanity.com/paper/2310.20150/) (2023)
- [Exploring Length Generalization in Large Language Models](https://mlsanity.com/paper/2207.04901/) (2022)
- [Transformer-Patcher: One Mistake worth One Neuron](https://mlsanity.com/paper/2301.09785/) (2023)

## Built on by (most cited first, 30 of 53)

- [Continual Learning of Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2404.16789/) (2024)
- [MIRIX: Multi-Agent Memory System for LLM-Based Agents](https://mlsanity.com/paper/2507.07957/) (2025)
- [A Survey of Context Engineering for Large Language Models](https://mlsanity.com/paper/2507.13334/) (2025)
- [Human-inspired Episodic Memory for Infinite Context LLMs](https://mlsanity.com/paper/2407.09450/) (2024)
- [Mem-α: Learning Memory Construction via Reinforcement Learning](https://mlsanity.com/paper/2509.25911/) (2025)
- [Machine Unlearning in Generative AI: A Survey](https://mlsanity.com/paper/2407.20516/) (2024)
- [Skill-Pro: Learning Reusable Skills from Experience via Non-Parametric PPO for LLM Agents](https://mlsanity.com/paper/2602.01869/) (2026)
- [Position: Episodic Memory is the Missing Piece for Long-Term LLM Agents](https://mlsanity.com/paper/2502.06975/) (2025)
- [Rethinking Memory in LLM based Agents: Representations, Operations, and Emerging Topics](https://mlsanity.com/paper/2505.00675/) (2025)
- [Needle in the Haystack for Memory Based Large Language Models](https://mlsanity.com/paper/2407.01437/) (2024)
- [From Storage to Experience: A Survey on the Evolution of LLM Agent Memory Mechanisms](https://mlsanity.com/paper/2605.06716/) (2026)
- [Neuron-Level Sequential Editing for Large Language Models](https://mlsanity.com/paper/2410.04045/) (2024)
- [Embodied Agents Meet Personalization: Investigating Challenges and Solutions Through the Lens of Memory Utilization](https://mlsanity.com/paper/2505.16348/) (2025)
- [Episodic Memories Generation and Evaluation Benchmark for Large Language Models](https://mlsanity.com/paper/2501.13121/) (2025)
- [Towards LifeSpan Cognitive Systems](https://mlsanity.com/paper/2409.13265/) (2024)
- [Human-inspired Perspectives: A Survey on AI Long-term Memory](https://mlsanity.com/paper/2411.00489/) (2024)
- [A Simple "Try Again" Can Elicit Multi-Turn LLM Reasoning](https://mlsanity.com/paper/2507.14295/) (2025)
- [Assessing Episodic Memory in LLMs with Sequence Order Recall Tasks](https://mlsanity.com/paper/2410.08133/) (2024)
- [AI Awareness](https://mlsanity.com/paper/2504.20084/) (2025)
- [Beyond Fact Retrieval: Episodic Memory for RAG with Generative Semantic Workspaces](https://mlsanity.com/paper/2511.07587/) (2025)
- [ExAnte: A Benchmark for Ex-Ante Inference in Large Language Models](https://mlsanity.com/paper/2505.19533/) (2025)
- [MindForge: Empowering Embodied Agents with Theory of Mind for Lifelong Cultural Learning](https://mlsanity.com/paper/2411.12977/) (2024)
- [MemGym: a Long-Horizon Memory Environment for LLM Agents](https://mlsanity.com/paper/2605.20833/) (2026)
- [Search-R3: Unifying Reasoning and Embedding in Large Language Models](https://mlsanity.com/paper/2510.07048/) (2025)
- [What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agents](https://mlsanity.com/paper/2607.08032/) (2026)
- [Scopes of Alignment](https://mlsanity.com/paper/2501.12405/) (2025)
- [Memory Helps, but Confabulation Misleads: Understanding Streaming Events in Videos with MLLMs](https://mlsanity.com/paper/2502.15457/) (2025)
- [Enhancing Reasoning with Collaboration and Memory](https://mlsanity.com/paper/2503.05944/) (2025)
- [Memorization and Knowledge Injection in Gated LLMs](https://mlsanity.com/paper/2504.21239/) (2025)
- [BMAM: Brain-inspired Multi-Agent Memory Framework](https://mlsanity.com/paper/2601.20465/) (2026)

## Abstract (arXiv)

Efficient and accurate updating of knowledge stored in Large Language Models (LLMs) is one of the most pressing research challenges today. This paper presents Larimar - a novel, brain-inspired architecture for enhancing LLMs with a distributed episodic memory. Larimar's memory allows for dynamic, one-shot updates of knowledge without the need for computationally expensive re-training or fine-tuning. Experimental results on multiple fact editing benchmarks demonstrate that Larimar attains accuracy comparable to most competitive baselines, even in the challenging sequential editing setup, but also excels in speed - yielding speed-ups of 8-10x depending on the base LLM - as well as flexibility due to the proposed architecture being simple, LLM-agnostic, and hence general. We further provide mechanisms for selective fact forgetting, information leakage prevention, and input context length generalization with Larimar and show their effectiveness. Our code is available at https://github.com/IBM/larimar