# Lightweight LLM Agent Memory with Small Language Models

arXiv: https://arxiv.org/abs/2604.07798 · HTML: https://mlsanity.com/paper/2604.07798/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in Agent memory systems, one of 16 directions in LLM agents and tool use.
- Builds on 16 of its 42 references that are in the map.
- First posted on arXiv 9 Apr 2026.

## Placed in

- LLM agents and tool use / [Agent memory systems](https://mlsanity.com/agents/agent-memory-systems/)

## Builds on (most cited first, 16 of 16)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://mlsanity.com/paper/2310.11511/) (2023)
- [A-MEM: Agentic Memory for LLM Agents](https://mlsanity.com/paper/2502.12110/) (2025)
- [Evaluating Very Long-Term Conversational Memory of LLM Agents](https://mlsanity.com/paper/2402.17753/) (2024)
- [A Survey on the Memory Mechanism of Large Language Model based Agents](https://mlsanity.com/paper/2404.13501/) (2024)
- [Understanding the planning of LLM agents: A survey](https://mlsanity.com/paper/2402.02716/) (2024)
- [Teaching Small Language Models to Reason](https://mlsanity.com/paper/2212.08410/) (2022)
- [MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents](https://mlsanity.com/paper/2503.01935/) (2025)
- [A Human-Inspired Reading Agent with Gist Memory of Very Long Contexts](https://mlsanity.com/paper/2402.09727/) (2024)
- [MemBench: Towards More Comprehensive Evaluation on the Memory of LLM-based Agents](https://mlsanity.com/paper/2506.21605/) (2025)
- [Small Language Model Can Self-correct](https://mlsanity.com/paper/2401.07301/) (2024)
- [Learning Global Hypothesis Space for Enhancing Synergistic Reasoning Chain](https://mlsanity.com/paper/2602.09794/) (2026)
- [Text summarization via global structure awareness](https://mlsanity.com/paper/2602.09821/) (2026)
- [Are Small Language Models Ready to Compete with Large Language Models for Practical Applications?](https://mlsanity.com/paper/2406.11402/) (2024)
- [Smaller Large Language Models Can Do Moral Self-Correction](https://mlsanity.com/paper/2410.23496/) (2024)

## Built on by (most cited first, 13 of 13)

- [Anticipate and Learn: Unleashing Idle-Time Compute in Proactive Agents](https://mlsanity.com/paper/2605.25971/) (2026)
- [Transforming External Knowledge into Triplets for Enhanced Retrieval in RAG of LLMs](https://mlsanity.com/paper/2604.12610/) (2026)
- [The Retriever Should Remember: Experience-Amortized Reranking for Long-Term Agent Memory](https://mlsanity.com/paper/2608.22767/) (2026)
- [CAP-CoT: Cycle Adversarial Prompt for Improving Chain of Thoughts in LLM Reasoning](https://mlsanity.com/paper/2604.23270/) (2026)
- [From Similarity to Structure: Training-free LLM Context Compression with Hybrid Graph Priors](https://mlsanity.com/paper/2604.23277/) (2026)
- [Making Prospective Memory SLM-Shaped: Typed Intention Stores for Small-Model Agents](https://mlsanity.com/paper/2609.01272/) (2026)
- [Memory Control Signals Emerge Before Action in Long Horizon Agents](https://mlsanity.com/paper/2609.27286/) (2026)
- [TF-SNO: Time-Frequency Gated Spectral Neural Operators for Learning Non-Stationary Partial Differential Equations](https://mlsanity.com/paper/2606.21189/) (2026)
- [State Compression in Two-Agent LLM Relays: A Closed-World Study of Constraint Preservation](https://mlsanity.com/paper/2607.18265/) (2026)
- [BudgetBench: A Budget-Tiered Protocol and Pilot Harness for Memory Strategy Evaluation in Local Large Language Model Agents](https://mlsanity.com/paper/2609.13149/) (2026)
- [When Should Agents Check External State? Budgeting Observations for Stored Intentions](https://mlsanity.com/paper/2609.37125/) (2026)
- [Learning to Retrieve Missing Evidence for Long-Term Memory QA](https://mlsanity.com/paper/2609.37443/) (2026)
- [MoR-MLLM: Mixture of Recursions for Efficient Multimodal Large Language Models](https://mlsanity.com/paper/2610.08830/) (2026)

## Abstract (arXiv)

Although LLM agents can leverage tools for complex tasks, they still need memory to maintain cross-turn consistency and accumulate reusable information in long-horizon interactions. However, retrieval-based external memory systems incur low online overhead but suffer from unstable accuracy due to limited query construction and candidate filtering. In contrast, many systems use repeated large-model calls for online memory operations, improving accuracy but accumulating latency over long interactions. We propose LightMem, a lightweight memory system for better agent memory driven by Small Language Models (SLMs). LightMem modularizes memory retrieval, writing, and long-term consolidation, and separates online processing from offline consolidation to enable efficient memory invocation under bounded compute. We organize memory into short-term memory (STM) for immediate conversational context, mid-term memory (MTM) for reusable interaction summaries, and long-term memory (LTM) for consolidated knowledge, and uses user identifiers to support independent retrieval and incremental maintenance in multi-user settings. Online, LightMem operates under a fixed retrieval budget and selects memories via a two-stage procedure: vector-based coarse retrieval followed by semantic consistency re-ranking. Offline, it abstracts reusable interaction evidence and incrementally integrates it into LTM. Experiments show consistent gains across model scales, with an average F1 improvement of about 2.5 over A-MEM on LoCoMo, while achieving higher efficiency and low median latency (83 ms for retrieval and 581 ms end-to-end).