# Toward a Theory of Hierarchical Memory for Language Agents

arXiv: https://arxiv.org/abs/2603.21564 · HTML: https://mlsanity.com/paper/2603.21564/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Agent memory systems, one of 16 directions in LLM agents and tool use.
- Builds on 15 of its 21 references that are in the map.
- First posted on arXiv 23 Mar 2026.

## Placed in

- LLM agents and tool use / [Agent memory systems](https://mlsanity.com/agents/agent-memory-systems/)

## Builds on (most cited first, 15 of 15)

- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [From Local to Global: A Graph RAG Approach to Query-Focused Summarization](https://mlsanity.com/paper/2404.16130/) (2024)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval](https://mlsanity.com/paper/2401.18059/) (2024)
- [Cognitive Architectures for Language Agents](https://mlsanity.com/paper/2309.02427/) (2023)
- [SimpleMem: Efficient Lifelong Memory for LLM Agents](https://mlsanity.com/paper/2601.02553/) (2026)
- [Walking Down the Memory Maze: Beyond Context Limit through Interactive Reading](https://mlsanity.com/paper/2310.05029/) (2023)
- [From Isolated Conversations to Hierarchical Schemas: Dynamic Tree Memory Representation for LLMs](https://mlsanity.com/paper/2410.14052/) (2024)
- [Agentic Memory: Learning Unified Long-Term and Short-Term Memory Management for Large Language Model Agents](https://mlsanity.com/paper/2601.01885/) (2026)
- [Hierarchical Memory for High-Efficiency Long-Term Reasoning in LLM Agents](https://mlsanity.com/paper/2507.22925/) (2025)
- [Beyond RAG for Agent Memory: Retrieval by Decoupling and Aggregation](https://mlsanity.com/paper/2602.02007/) (2026)
- [MemEngine: A Unified and Modular Library for Developing Advanced Memory of LLM-based Agents](https://mlsanity.com/paper/2505.02099/) (2025)
- [StackPlanner: A Centralized Hierarchical Multi-Agent System with Task-Experience Memory Management](https://mlsanity.com/paper/2601.05890/) (2026)
- [MemoBrain: Executive Memory as an Agentic Brain for Reasoning](https://mlsanity.com/paper/2601.08079/) (2026)
- [InfiAgent: An Infinite-Horizon Framework for General-Purpose Autonomous Agents](https://mlsanity.com/paper/2601.03204/) (2026)

## Built on by (most cited first, 5 of 5)

- [From Signals to Structure: How Memory Architecture Drives Language Emergence in LLM Agents](https://mlsanity.com/paper/2607.00233/) (2026)
- [Why Solve It Twice? Hierarchical Accumulation of Skills for Transfer-Efficient ML Engineering](https://mlsanity.com/paper/2606.30911/) (2026)
- [Memory Is Communication: The Frontier Between Remembering and Signaling](https://mlsanity.com/paper/2608.17053/) (2026)
- [Toward Personal Intelligence Through Cooperative Observation](https://mlsanity.com/paper/2608.17128/) (2026)
- [Dude, Where's My State? Execution Information Requirements for Stateful Agents](https://mlsanity.com/paper/2609.32687/) (2026)

## Abstract (arXiv)

Many recent long-context and agentic systems address context-length limitations by adding hierarchical memory: they extract atomic units from raw data, build multi-level representatives by grouping and compression, and traverse this structure to retrieve content under a token budget. Despite recurring implementations, there is no shared formalism for comparing design choices. We propose a unifying theory in terms of three operators. Extraction (α) maps raw data to atomic information units; coarsening (C = (π, ρ)) partitions units and assigns a representative to each group; and traversal (τ) selects which units to include in context given a query and budget. We identify a self-sufficiency spectrum for the representative function ρ and show how it constrains viable retrieval strategies (a coarsening-traversal coupling). Finally, we instantiate the decomposition on eleven existing systems spanning document hierarchies, conversational memory, and agent execution traces, showcasing its generality.