# Hierarchical Memory for High-Efficiency Long-Term Reasoning in LLM Agents

arXiv: https://arxiv.org/abs/2507.22925 · HTML: https://mlsanity.com/paper/2507.22925/

## Key facts

- Cited by 62 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 16 of its 23 references that are in the map.
- First posted on arXiv 23 Jul 2025.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)
- LLM agents and tool use / [Agent memory systems](https://mlsanity.com/agents/agent-memory-systems/)

## Builds on (most cited first, 16 of 16)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [A-MEM: Agentic Memory for LLM Agents](https://mlsanity.com/paper/2502.12110/) (2025)
- [Evaluating Very Long-Term Conversational Memory of LLM Agents](https://mlsanity.com/paper/2402.17753/) (2024)
- [A Survey on the Memory Mechanism of Large Language Model based Agents](https://mlsanity.com/paper/2404.13501/) (2024)
- [Understanding the planning of LLM agents: A survey](https://mlsanity.com/paper/2402.02716/) (2024)
- [The Faiss library](https://mlsanity.com/paper/2401.08281/) (2024)
- [Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Security](https://mlsanity.com/paper/2401.05459/) (2024)
- [A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systems](https://mlsanity.com/paper/2402.18013/) (2024)
- [A Human-Inspired Reading Agent with Gist Memory of Very Long Contexts](https://mlsanity.com/paper/2402.09727/) (2024)
- [SCM: Enhancing Large Language Model with Self-Controlled Memory Framework](https://mlsanity.com/paper/2304.13343/) (2023)
- [MemInsight: Autonomous Memory Augmentation for LLM Agents](https://mlsanity.com/paper/2503.21760/) (2025)
- [LLM With Tools: A Survey](https://mlsanity.com/paper/2409.18807/) (2024)

## Built on by (most cited first, 30 of 62)

- [MemEvolve: Meta-Evolution of Agent Memory Systems](https://mlsanity.com/paper/2512.18746/) (2025)
- [EverMemOS: A Self-Organizing Memory Operating System for Structured Long-Horizon Reasoning](https://mlsanity.com/paper/2601.02163/) (2026)
- [A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents](https://mlsanity.com/paper/2602.06052/) (2026)
- [HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understanding](https://mlsanity.com/paper/2601.14724/) (2026)
- [LEGOMem: Modular Procedural Memory for Multi-agent LLM Systems for Workflow Automation](https://mlsanity.com/paper/2510.04851/) (2025)
- [Graph-based Agent Memory: Taxonomy, Techniques, and Applications](https://mlsanity.com/paper/2602.05665/) (2026)
- [Preference-Aware Memory Update for Long-Term LLM Agents](https://mlsanity.com/paper/2510.09720/) (2025)
- [Self-Improvements in Modern Agentic Systems: A Survey](https://mlsanity.com/paper/2607.13104/) (2026)
- [Context Engineering 2.0: The Context of Context Engineering](https://mlsanity.com/paper/2510.26493/) (2025)
- [Memory Matters More: Event-Centric Memory as a Logic Map for Agent Searching and Reasoning](https://mlsanity.com/paper/2601.04726/) (2026)
- [APEX-MEM: Agentic Semi-Structured Memory with Temporal Reasoning for Long-Term Conversational AI](https://mlsanity.com/paper/2604.14362/) (2026)
- [Toward Efficient Agents: Memory, Tool learning, and Planning](https://mlsanity.com/paper/2601.14192/) (2026)
- [Beyond Dialogue Time: Temporal Semantic Memory for Personalized LLM Agents](https://mlsanity.com/paper/2601.07468/) (2026)
- [AI Meets Brain: Memory Systems from Cognitive Neuroscience to Autonomous Agents](https://mlsanity.com/paper/2512.23343/) (2025)
- [Evaluating Memory Structure in LLM Agents](https://mlsanity.com/paper/2602.11243/) (2026)
- [Memory in the LLM Era: Modular Architectures and Strategies in a Unified Framework](https://mlsanity.com/paper/2604.01707/) (2026)
- [Beyond Pipelines: A Survey of the Paradigm Shift toward Model-Native Agentic AI](https://mlsanity.com/paper/2510.16720/) (2025)
- [Semantic XPath: Structured Agentic Memory Access for Conversational AI](https://mlsanity.com/paper/2603.01160/) (2026)
- [Agentic Trading: When LLM Agents Meet Financial Markets](https://mlsanity.com/paper/2605.19337/) (2026)
- [Learning from Supervision with Semantic and Episodic Memory: A Reflective Approach to Agent Adaptation](https://mlsanity.com/paper/2510.19897/) (2025)
- [ReflecSched: Solving Dynamic Flexible Job-Shop Scheduling via LLM-Powered Hierarchical Reflection](https://mlsanity.com/paper/2508.01724/) (2025)
- [See, Think, Act: Online Shopper Behavior Simulation with VLM Agents](https://mlsanity.com/paper/2510.19245/) (2025)
- [Toward a Theory of Hierarchical Memory for Language Agents](https://mlsanity.com/paper/2603.21564/) (2026)
- [HiGMem: A Hierarchical and LLM-Guided Memory System for Long-Term Conversational Agents](https://mlsanity.com/paper/2604.18349/) (2026)
- [Mitigating Provenance-Role Collapse in Long-Term Agents via Typed Memory Representation](https://mlsanity.com/paper/2605.25869/) (2026)
- [LazyMem: Retrieve Broadly, Construct Selectively for Efficient Long-Term Agent Memory](https://mlsanity.com/paper/2607.22690/) (2026)
- [ES-Mem: Event Segmentation-Based Memory for Long-Term Dialogue Agents](https://mlsanity.com/paper/2601.07582/) (2026)
- [CarePilot: A Multi-Agent Framework for Long-Horizon Computer Task Automation in Healthcare](https://mlsanity.com/paper/2603.24157/) (2026)
- [Learning to Retrieve: Dual-Level Long-Term Memory for Text-to-SQL Agents](https://mlsanity.com/paper/2606.00547/) (2026)
- [From Atom to Community: Structured and Evolving Agent Memory for User Behavior Modeling](https://mlsanity.com/paper/2601.16872/) (2026)

## Abstract (arXiv)

Long-term memory is one of the key factors influencing the reasoning capabilities of Large Language Model Agents (LLM Agents). Incorporating a memory mechanism that effectively integrates past interactions can significantly enhance decision-making and contextual coherence of LLM Agents. While recent works have made progress in memory storage and retrieval, such as encoding memory into dense vectors for similarity-based search or organizing knowledge in the form of graph, these approaches often fall short in structured memory organization and efficient retrieval. To address these limitations, we propose a Hierarchical Memory (H-MEM) architecture for LLM Agents that organizes and updates memory in a multi-level fashion based on the degree of semantic abstraction. Each memory vector is embedded with a positional index encoding pointing to its semantically related sub-memories in the next layer. During the reasoning phase, an index-based routing mechanism enables efficient, layer-by-layer retrieval without performing exhaustive similarity computations. We evaluate our method on five task settings from the LoCoMo dataset. Experimental results show that our approach consistently outperforms five baseline methods, demonstrating its effectiveness in long-term dialogue scenarios.