mlsanityData to 14 Oct 2026

A Grounded Memory System For Smart Personal Assistants

arXiv 2505.06328 ↗First posted 9 May 2025cs.AIcited by 5

On the map: LLM agents and tool use → Agent memory systems

Abstract (arXiv)

A wide variety of agentic AI applications - ranging from cognitive assistants for dementia patients to robotics - demand a robust memory system grounded in reality. In this paper, we propose such a memory system consisting of three components. First, we combine Vision Language Models for image captioning and entity disambiguation with Large Language Models for consistent information extraction during perception. Second, the extracted information is represented in a memory consisting of a knowledge graph enhanced by vector embeddings to efficiently manage relational information. Third, we combine semantic search and graph query generation for question answering via Retrieval Augmented Generation. We illustrate the system's working and potential using a real-world example.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Agent memory systems, one of 16 directions in LLM agents and tool use. Builds on 15 of its 21 references that are in the map. First posted on arXiv 9 May 2025.

Builds on

15 of 21 references.

References · 15

  1. 2024From Local to Global: A Graph RAG Approach to Query-Focused Summarizationcited by 1,453
  2. 2020Knowledge Graphscited by 621
  3. 20193D Scene Graph: A Structure for Unified Semantics, 3D Space, and Cameracited by 281
  4. 2024HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Modelscited by 281
  5. 2024VideoAgent: A Memory-augmented Multimodal Agent for Video Understandingcited by 229
  6. 20203D Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humanscited by 144
  7. 20193-D Scene Graph: A Sparse and Semantic Representation of Physical Environments for Intelligent Agentscited by 74
  8. 2022TASKOGRAPHY: Evaluating robot task planning over large 3D scene graphscited by 59
  9. 2023LifelongMemory: Leveraging LLMs for Answering Queries in Long-form Egocentric Videoscited by 51
  10. 2024Embodied-RAG: General Non-parametric Embodied Memory for Retrieval and Generationcited by 44
  11. 2024AMEGO: Active Memory from long EGOcentric videoscited by 36
  12. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 36
3 more
  1. 2024OmniQuery: Contextually Augmenting Captured Multimodal Memory to Enable Personal Question Answeringcited by 10
  2. 2024Tulip Agent -- Enabling LLM-Based Agents to Solve Tasks Using Large Tool Librariescited by 7
  3. 2025MemPal: Leveraging Multimodal AI and LLMs for Voice-Activated Object Retrieval in Homes of Older Adultscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

  1. 2025A Survey of Context Engineering for Large Language Modelscited by 126
  2. 2025Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with Long-Term Memorycited by 97
  3. 2025O-Mem: Omni Memory System for Personalized, Long Horizon, Self-Evolving Agentscited by 32
  4. 2026Toward Personalized LLM-Powered Agents: Foundations, Evaluation, and Future Directionscited by 14
  5. 2026SF-AMS: Strategic Forgetting for Structured Memory in LLM Agentcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025AUGUSTUS: An LLM-Driven Multimodal Agent System with Contextualized User Memory
  2. 2026EmbodiedLGR: Integrating Lightweight Graph Representation and Retrieval for Semantic-Spatial Memory in Robotic Agents
  3. 2026MemORAI: Memory Organization and Retrieval via Adaptive Graph Intelligence for LLM Conversational Agents
  4. 2025PersonalAI: A Systematic Comparison of Knowledge Graph Storage and Retrieval Approaches for Personalized LLM agents
  5. 2026MemMachine: A Ground-Truth-Preserving Memory System for Personalized AI Agents
  6. 2025MemInsight: Autonomous Memory Augmentation for LLM Agents
  7. 2026Memory-Augmented Vision-Language Agents for Persistent and Semantically Consistent Object Captioning
  8. 2026Memanto: Typed Semantic Memory with Information-Theoretic Retrieval for Long-Horizon Agents