# M+: Extending MemoryLLM with Scalable Long-Term Memory

arXiv: https://arxiv.org/abs/2502.00592 · HTML: https://mlsanity.com/paper/2502.00592/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 39 references that are in the map.
- First posted on arXiv 1 Feb 2025.

## Builds on (most cited first, 30 of 30)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://mlsanity.com/paper/2308.14508/) (2023)
- [MemGPT: Towards LLMs as Operating Systems](https://mlsanity.com/paper/2310.08560/) (2023)
- [The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale](https://mlsanity.com/paper/2406.17557/) (2024)
- [SnapKV: LLM Knows What You are Looking for Before Generation](https://mlsanity.com/paper/2404.14469/) (2024)
- [H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models](https://mlsanity.com/paper/2306.14048/) (2023)
- [Generalization through Memorization: Nearest Neighbor Language Models](https://mlsanity.com/paper/1911.00172/) (2019)
- [Training Large Language Models to Reason in a Continuous Latent Space](https://mlsanity.com/paper/2412.06769/) (2024)
- [MemoryBank: Enhancing Large Language Models with Long-Term Memory](https://mlsanity.com/paper/2305.10250/) (2023)
- [∞Bench: Extending Long Context Evaluation Beyond 100K Tokens](https://mlsanity.com/paper/2402.13718/) (2024)
- [HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models](https://mlsanity.com/paper/2405.14831/) (2024)
- [Memorizing Transformers](https://mlsanity.com/paper/2203.08913/) (2022)
- [A Survey of Human-in-the-loop for Machine Learning](https://mlsanity.com/paper/2108.00941/) (2021)
- [Recurrent Memory Transformer](https://mlsanity.com/paper/2207.06881/) (2022)
- [In-context Autoencoder for Context Compression in a Large Language Model](https://mlsanity.com/paper/2307.06945/) (2023)
- [Augmenting Language Models with Long-Term Memory](https://mlsanity.com/paper/2306.07174/) (2023)
- [Data Engineering for Scaling Language Models to 128K Context](https://mlsanity.com/paper/2402.10171/) (2024)
- [ChatDB: Augmenting LLMs with Databases as Their Symbolic Memory](https://mlsanity.com/paper/2306.03901/) (2023)
- [Scaling Transformer to 1M tokens and beyond with RMT](https://mlsanity.com/paper/2304.11062/) (2023)
- [MEMORYLLM: Towards Self-Updatable Large Language Models](https://mlsanity.com/paper/2402.04624/) (2024)
- [RecurrentGPT: Interactive Generation of (Arbitrarily) Long Text](https://mlsanity.com/paper/2305.13304/) (2023)
- [Memory3: Language Modeling with Explicit Memory](https://mlsanity.com/paper/2407.01178/) (2024)
- [MemLLM: Finetuning LLMs to Use An Explicit Read-Write Memory](https://mlsanity.com/paper/2404.11672/) (2024)
- [Who's Who: Large Language Models Meet Knowledge Conflicts in Practice](https://mlsanity.com/paper/2410.15737/) (2024)
- [Large Scale Knowledge Washing](https://mlsanity.com/paper/2405.16720/) (2024)
- [Towards LifeSpan Cognitive Systems](https://mlsanity.com/paper/2409.13265/) (2024)
- [MinPrompt: Graph-based Minimal Prompt Data Augmentation for Few-shot Question Answering](https://mlsanity.com/paper/2310.05007/) (2023)
- [Tiny Transformers Excel at Sentence Compression](https://mlsanity.com/paper/2410.23510/) (2024)
- [An Enhanced Text Compression Approach Using Transformer-based Language Models](https://mlsanity.com/paper/2412.15250/) (2024)
- [MemoryPrompt: A Light Wrapper to Improve Context Tracking in Pre-trained Language Models](https://mlsanity.com/paper/2402.15268/) (2024)

## Built on by (most cited first, 30 of 54)

- [Memory in the Age of AI Agents](https://mlsanity.com/paper/2512.13564/) (2025)
- [ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory](https://mlsanity.com/paper/2509.25140/) (2025)
- [Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions](https://mlsanity.com/paper/2507.05257/) (2025)
- [MIRIX: Multi-Agent Memory System for LLM-Based Agents](https://mlsanity.com/paper/2507.07957/) (2025)
- [Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systems](https://mlsanity.com/paper/2504.01990/) (2025)
- [MemGen: Weaving Generative Latent Memory for Self-Evolving Agents](https://mlsanity.com/paper/2509.24704/) (2025)
- [Mem-α: Learning Memory Construction via Reinforcement Learning](https://mlsanity.com/paper/2509.25911/) (2025)
- [MemSearcher: Training LLMs to Reason, Search and Manage Memory via End-to-End Reinforcement Learning](https://mlsanity.com/paper/2511.02805/) (2025)
- [DeltaEvolve: Accelerating Scientific Discovery through Momentum-Driven Evolution](https://mlsanity.com/paper/2602.02919/) (2026)
- [A Survey on Agentic Multimodal Large Language Models](https://mlsanity.com/paper/2510.10991/) (2025)
- [A Comprehensive Survey on Agent Skills: Taxonomy, Techniques, and Applications](https://mlsanity.com/paper/2605.07358/) (2026)
- [WebCoach: Self-Evolving Web Agents with Cross-Session Memory Guidance](https://mlsanity.com/paper/2511.12997/) (2025)
- [Towards General Continuous Memory for Vision-Language Models](https://mlsanity.com/paper/2505.17670/) (2025)
- [According to Me: Long-Term Personalized Referential Memory QA](https://mlsanity.com/paper/2603.01990/) (2026)
- [δ-mem: Efficient Online Memory for Large Language Models](https://mlsanity.com/paper/2605.12357/) (2026)
- [Evaluating Memory Structure in LLM Agents](https://mlsanity.com/paper/2602.11243/) (2026)
- [Memex(RL): Scaling Long-Horizon LLM Agents via Indexed Experience Memory](https://mlsanity.com/paper/2603.04257/) (2026)
- [M3HF: Multi-agent Reinforcement Learning from Multi-phase Human Feedback of Mixed Quality](https://mlsanity.com/paper/2503.02077/) (2025)
- [Artificial Hippocampus Networks for Efficient Long-Context Modeling](https://mlsanity.com/paper/2510.07318/) (2025)
- [Memory-Augmented Transformers: A Systematic Review from Neuroscience Principles to Enhanced Model Architectures](https://mlsanity.com/paper/2508.10824/) (2025)
- [LatentEvolve: Self-Evolving Test-Time Scaling in Latent Space](https://mlsanity.com/paper/2509.24771/) (2025)
- [Beyond Pipelines: A Survey of the Paradigm Shift toward Model-Native Agentic AI](https://mlsanity.com/paper/2510.16720/) (2025)
- [Bidirectional LMs are Better Knowledge Memorizers? A Benchmark for Real-world Knowledge Injection](https://mlsanity.com/paper/2505.12306/) (2025)
- [TRUSTMEM: Learning Trustworthy Memory Consolidation for LLM Agents with Long-Term Memory](https://mlsanity.com/paper/2606.25161/) (2026)
- [MemSearch-o1: Empowering Large Language Models with Reasoning-Aligned Memory Growth in Agentic Search](https://mlsanity.com/paper/2604.17265/) (2026)
- [G-MemLLM: Gated Latent Memory Augmentation for Long-Context Reasoning in Large Language Models](https://mlsanity.com/paper/2602.00015/) (2026)
- [ElasticMem: Latent Memory as a Learnable Resource for LLM Agents](https://mlsanity.com/paper/2605.30690/) (2026)
- [DynamicMem: A Long-Horizon Memory Benchmark in Real-World Settings](https://mlsanity.com/paper/2606.22877/) (2026)
- [A-TMA: Decoupling State-Aware Memory Failures in Long-Term Agent Memory](https://mlsanity.com/paper/2607.01935/) (2026)
- [The Narrative Continuity Test: A Conceptual Framework for Evaluating Identity Persistence in AI Systems](https://mlsanity.com/paper/2510.24831/) (2025)

## Abstract (arXiv)

Equipping large language models (LLMs) with latent-space memory has attracted increasing attention as they can extend the context window of existing language models. However, retaining information from the distant past remains a challenge. For example, MemoryLLM (Wang et al., 2024a), as a representative work with latent-space memory, compresses past information into hidden states across all layers, forming a memory pool of 1B parameters. While effective for sequence lengths up to 16k tokens, it struggles to retain knowledge beyond 20k tokens. In this work, we address this limitation by introducing M+, a memory-augmented model based on MemoryLLM that significantly enhances long-term information retention. M+ integrates a long-term memory mechanism with a co-trained retriever, dynamically retrieving relevant information during text generation. We evaluate M+ on diverse benchmarks, including long-context understanding and knowledge retention tasks. Experimental results show that M+ significantly outperforms MemoryLLM and recent strong baselines, extending knowledge retention from under 20k to over 160k tokens with similar GPU memory overhead. We open-source our code at https://github.com/wangyu-ustc/MemoryLLM