# HiGMem: A Hierarchical and LLM-Guided Memory System for Long-Term Conversational Agents

arXiv: https://arxiv.org/abs/2604.18349 · HTML: https://mlsanity.com/paper/2604.18349/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Agent memory systems, one of 16 directions in LLM agents and tool use.
- Builds on 16 of its 18 references that are in the map.
- First posted on arXiv 20 Apr 2026.

## Placed in

- LLM agents and tool use / [Agent memory systems](https://mlsanity.com/agents/agent-memory-systems/)

## Builds on (most cited first, 16 of 16)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [MemGPT: Towards LLMs as Operating Systems](https://mlsanity.com/paper/2310.08560/) (2023)
- [A-MEM: Agentic Memory for LLM Agents](https://mlsanity.com/paper/2502.12110/) (2025)
- [Evaluating Very Long-Term Conversational Memory of LLM Agents](https://mlsanity.com/paper/2402.17753/) (2024)
- [A Survey on the Memory Mechanism of Large Language Model based Agents](https://mlsanity.com/paper/2404.13501/) (2024)
- [RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval](https://mlsanity.com/paper/2401.18059/) (2024)
- [Beyond Goldfish Memory: Long-Term Open-Domain Conversation](https://mlsanity.com/paper/2107.07567/) (2021)
- [A Human-Inspired Reading Agent with Gist Memory of Very Long Contexts](https://mlsanity.com/paper/2402.09727/) (2024)
- [Hierarchical Memory for High-Efficiency Long-Term Reasoning in LLM Agents](https://mlsanity.com/paper/2507.22925/) (2025)
- [Conversation Chronicles: Towards Diverse Temporal and Relational Dynamics in Multi-Session Conversations](https://mlsanity.com/paper/2310.13420/) (2023)
- [daDPO: Distribution-Aware DPO for Distilling Conversational Abilities](https://mlsanity.com/paper/2506.15717/) (2025)
- [ReSURE: Regularizing Supervision Unreliability for Multi-turn Dialogue Fine-tuning](https://mlsanity.com/paper/2508.19996/) (2025)

## Built on by (most cited first, 5 of 5)

- [MemChain: Learning Interpretable Memory Traces for Memory-Augmented LLM Agents](https://mlsanity.com/paper/2607.24097/) (2026)
- [When Users Don't Ask: Benchmarking Context-Driven Memory Retrieval in Conversational Agents](https://mlsanity.com/paper/2609.03467/) (2026)
- [CABLE: Extending the Reach of Memory Retrieval via Complementary Antecedent-Based Linking and Expansion](https://mlsanity.com/paper/2608.17911/) (2026)
- [JustMem: Just-Enough Memory Access for Long-Term Conversations](https://mlsanity.com/paper/2609.19877/) (2026)
- [Memory Control Signals Emerge Before Action in Long Horizon Agents](https://mlsanity.com/paper/2609.27286/) (2026)

## Abstract (arXiv)

Long-term conversational large language model (LLM) agents require memory systems that can recover relevant evidence from historical interactions without overwhelming the answer stage with irrelevant context. However, existing memory systems, including hierarchical ones, still often rely solely on vector similarity for retrieval. It tends to produce bloated evidence sets: adding many superficially similar dialogue turns yields little additional recall, but lowers retrieval precision, increases answer-stage context cost, and makes retrieved memories harder to inspect and manage. To address this, we propose HiGMem (Hierarchical and LLM-Guided Memory System), a two-level event-turn memory system that allows LLMs to use event summaries as semantic anchors to predict which related turns are worth reading. This allows the model to inspect high-level event summaries first and then focus on a smaller set of potentially useful turns, providing a concise and reliable evidence set through reasoning, while avoiding the retrieval overhead that would be excessively high compared to vector retrieval. On the LoCoMo10 benchmark, HiGMem achieves the best F1 on four of five question categories and improves adversarial F1 from 0.54 to 0.78 over A-Mem, while retrieving an order of magnitude fewer turns. Code is publicly available at https://github.com/ZeroLoss-Lab/HiGMem.