# MemVerse: Multimodal Memory for Lifelong Learning Agents

arXiv: https://arxiv.org/abs/2512.03627 · HTML: https://mlsanity.com/paper/2512.03627/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 76 references that are in the map.
- First posted on arXiv 3 Dec 2025.

## Builds on (most cited first, 30 of 41)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [MemGPT: Towards LLMs as Operating Systems](https://mlsanity.com/paper/2310.08560/) (2023)
- [A-MEM: Agentic Memory for LLM Agents](https://mlsanity.com/paper/2502.12110/) (2025)
- [Evaluating Very Long-Term Conversational Memory of LLM Agents](https://mlsanity.com/paper/2402.17753/) (2024)
- [LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory](https://mlsanity.com/paper/2410.10813/) (2024)
- [A Survey on the Memory Mechanism of Large Language Model based Agents](https://mlsanity.com/paper/2404.13501/) (2024)
- [InternVideo: General Video Foundation Models via Generative and Discriminative Learning](https://mlsanity.com/paper/2212.03191/) (2022)
- [Zep: A Temporal Knowledge Graph Architecture for Agent Memory](https://mlsanity.com/paper/2501.13956/) (2025)
- [X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrieval](https://mlsanity.com/paper/2207.07285/) (2022)
- [FireAct: Toward Language Agent Fine-tuning](https://mlsanity.com/paper/2310.05915/) (2023)
- [MemNet: A Persistent Memory Network for Image Restoration](https://mlsanity.com/paper/1708.02209/) (2017)
- [Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learning](https://mlsanity.com/paper/2508.19828/) (2025)
- [VideoAgent: A Memory-augmented Multimodal Agent for Video Understanding](https://mlsanity.com/paper/2403.11481/) (2024)
- [Unmasked Teacher: Towards Training-Efficient Video Foundation Models](https://mlsanity.com/paper/2303.16058/) (2023)
- [MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognition](https://mlsanity.com/paper/2201.08383/) (2022)
- [MIRIX: Multi-Agent Memory System for LLM-Based Agents](https://mlsanity.com/paper/2507.07957/) (2025)
- [VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset](https://mlsanity.com/paper/2305.18500/) (2023)
- [MM-Embed: Universal Multimodal Retrieval with Multimodal LLMs](https://mlsanity.com/paper/2411.02571/) (2024)
- [Embedding-based Retrieval in Facebook Search](https://mlsanity.com/paper/2006.11632/) (2020)
- [mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Video](https://mlsanity.com/paper/2302.00402/) (2023)
- [Neural Episodic Control](https://mlsanity.com/paper/1703.01988/) (2017)
- [Hello Again! LLM-powered Personalized Agent for Long-term Dialogue](https://mlsanity.com/paper/2406.05925/) (2024)
- [MemoRAG: Boosting Long Context Processing with Global Memory-Enhanced Retrieval Augmentation](https://mlsanity.com/paper/2409.05591/) (2024)
- [Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models](https://mlsanity.com/paper/2411.14432/) (2024)
- [MemGen: Weaving Generative Latent Memory for Self-Evolving Agents](https://mlsanity.com/paper/2509.24704/) (2025)
- [MegaPairs: Massive Data Synthesis For Universal Multimodal Retrieval](https://mlsanity.com/paper/2412.14475/) (2024)
- [FG-CLIP: Fine-Grained Visual and Textual Alignment](https://mlsanity.com/paper/2505.05071/) (2025)

## Built on by (most cited first, 30 of 35)

- [Memory in the Age of AI Agents](https://mlsanity.com/paper/2512.13564/) (2025)
- [XSkill: Continual Learning from Experience and Skills in Multimodal Agents](https://mlsanity.com/paper/2603.12056/) (2026)
- [Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering](https://mlsanity.com/paper/2604.08224/) (2026)
- [Hindsight is 20/20: Building Agent Memory that Retains, Recalls, and Reflects](https://mlsanity.com/paper/2512.12818/) (2025)
- [A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents](https://mlsanity.com/paper/2602.06052/) (2026)
- [Governing Evolving Memory in LLM Agents: Risks, Mechanisms, and the Stability and Safety Governed Memory (SSGM) Framework](https://mlsanity.com/paper/2603.11768/) (2026)
- [EvolveMem:Self-Evolving Memory Architecture via AutoResearch for LLM Agents](https://mlsanity.com/paper/2605.13941/) (2026)
- [TeleMem: Building Long-Term and Multimodal Memory for Agentic AI](https://mlsanity.com/paper/2601.06037/) (2025)
- [Omni-SimpleMem: Autoresearch-Guided Discovery of Lifelong Multimodal Agent Memory](https://mlsanity.com/paper/2604.01007/) (2026)
- [AI Meets Brain: Memory Systems from Cognitive Neuroscience to Autonomous Agents](https://mlsanity.com/paper/2512.23343/) (2025)
- [MementoGUI: Learning Agentic Multimodal Memory Control for Long-Horizon GUI Agents](https://mlsanity.com/paper/2605.18652/) (2026)
- [Position: Modular Memory is the Key to Continual Learning Agents](https://mlsanity.com/paper/2603.01761/) (2026)
- [One Model, All Roles: Multi-Turn, Multi-Agent Self-Play Reinforcement Learning for Conversational Social Intelligence](https://mlsanity.com/paper/2602.03109/) (2026)
- [SMMBench: A Benchmark for Source-Distributed Multimodal Agent Memory](https://mlsanity.com/paper/2605.15710/) (2026)
- [M3Exam: Benchmarking Multimodal Memory for Realistic User-Agent Interactions](https://mlsanity.com/paper/2606.07402/) (2026)
- [Hierarchical Memory Orchestration for Personalized Persistent Agents](https://mlsanity.com/paper/2604.01670/) (2026)
- [Persistent Visual Memory: Sustaining Perception for Deep Generation in LVLMs](https://mlsanity.com/paper/2605.00814/) (2026)
- [V-Mem: Modality-Routed Retrieval for Long-Term Multimodal Agentic Memory](https://mlsanity.com/paper/2608.01543/) (2026)
- [HIMM: Human-Inspired Long-Term Memory Modeling for Embodied Exploration and Question Answering](https://mlsanity.com/paper/2602.15513/) (2026)
- [FSFM: A Biologically-Inspired Framework for Selective Forgetting of Agent Memory](https://mlsanity.com/paper/2604.20300/) (2026)
- [Mem-π: Adaptive Memory through Learning When and What to Generate](https://mlsanity.com/paper/2605.21463/) (2026)
- [Mi-Memory: A Lifecycle Memory Framework for Personal AI](https://mlsanity.com/paper/2607.18975/) (2026)
- [Personal Visual Memory from Explicit and Implicit Evidence](https://mlsanity.com/paper/2605.28806/) (2026)
- [Do Agents Dream of False Memories? Black-box Visual Attacks on Long-term Memory in Multimodal AI Agents](https://mlsanity.com/paper/2607.15657/) (2026)
- [MIND: Multi-rationale INtegrated Discriminative Reasoning Framework for Multi-modal Large Models](https://mlsanity.com/paper/2512.05530/) (2025)
- [Learning to Learn from Multimodal Experience](https://mlsanity.com/paper/2605.16857/) (2026)
- [Task-Focused Memorization for Multimodal Agents](https://mlsanity.com/paper/2605.31075/) (2026)
- [HiMe: Hierarchical Embodied Memory for Long-Horizon Vision-Language-Action Control](https://mlsanity.com/paper/2607.03449/) (2026)
- [TransMem: Transforming Hidden States into Memory for Large Language Models](https://mlsanity.com/paper/2607.29032/) (2026)
- [Retrieved but Not Delivered: Multimodal Memory Delivery for Long-Term Agents](https://mlsanity.com/paper/2609.32590/) (2026)

## Abstract (arXiv)

Despite rapid progress in large-scale language and vision models, AI agents still suffer from a fundamental limitation: they cannot remember. Without reliable memory, agents catastrophically forget past experiences, struggle with long-horizon reasoning, and fail to operate coherently in multimodal or interactive environments. We introduce MemVerse, a model-agnostic, plug-and-play memory framework that bridges fast parametric recall with hierarchical retrieval-based memory, enabling scalable and adaptive multimodal intelligence. MemVerse maintains short-term memory for recent context while transforming raw multimodal experiences into structured long-term memories organized as hierarchical knowledge graphs. This design supports continual consolidation, adaptive forgetting, and bounded memory growth. To handle real-time demands, MemVerse introduces a periodic distillation mechanism that compresses essential knowledge from long-term memory into the parametric model, allowing fast, differentiable recall while preserving interpretability. Extensive experiments demonstrate that MemVerse significantly improves multimodal reasoning and continual learning efficiency, empowering agents to remember, adapt, and reason coherently across extended interactions.