# Mem-Gallery: Benchmarking Multimodal Long-Term Conversational Memory for MLLM Agents

arXiv: https://arxiv.org/abs/2601.03515 · HTML: https://mlsanity.com/paper/2601.03515/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 27 of its 46 references that are in the map.
- First posted on arXiv 7 Jan 2026.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)
- LLM agents and tool use / [Agent memory systems](https://mlsanity.com/agents/agent-memory-systems/)

## Builds on (most cited first, 27 of 27)

- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [MemGPT: Towards LLMs as Operating Systems](https://mlsanity.com/paper/2310.08560/) (2023)
- [A-MEM: Agentic Memory for LLM Agents](https://mlsanity.com/paper/2502.12110/) (2025)
- [Evaluating Very Long-Term Conversational Memory of LLM Agents](https://mlsanity.com/paper/2402.17753/) (2024)
- [LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory](https://mlsanity.com/paper/2410.10813/) (2024)
- [A Survey on the Memory Mechanism of Large Language Model based Agents](https://mlsanity.com/paper/2404.13501/) (2024)
- [From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge](https://mlsanity.com/paper/2411.16594/) (2024)
- [Memory in the Age of AI Agents](https://mlsanity.com/paper/2512.13564/) (2025)
- [Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions](https://mlsanity.com/paper/2507.05257/) (2025)
- [Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learning](https://mlsanity.com/paper/2508.19828/) (2025)
- [MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text](https://mlsanity.com/paper/2210.02928/) (2022)
- [LightMem: Lightweight and Efficient Memory-Augmented Generation](https://mlsanity.com/paper/2510.18866/) (2025)
- [Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Do](https://mlsanity.com/paper/2103.11790/) (2021)
- [GME: Improving Universal Multimodal Retrieval by Multimodal LLMs](https://mlsanity.com/paper/2412.16855/) (2024)
- [Evo-Memory: Benchmarking LLM Agent Test-time Learning with Self-Evolving Memory](https://mlsanity.com/paper/2511.20857/) (2025)
- [A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systems](https://mlsanity.com/paper/2504.09037/) (2025)
- [Long Time No See! Open-Domain Conversation with Long-Term Persona Memory](https://mlsanity.com/paper/2203.05797/) (2022)
- [How Memory Management Impacts LLM Agents: An Empirical Study of Experience-Following Behavior](https://mlsanity.com/paper/2505.16067/) (2025)
- [MMDialog: A Large-scale Multi-turn Dialogue Dataset Towards Multi-modal Open-domain Conversation](https://mlsanity.com/paper/2211.05719/) (2022)
- [DialogBench: Evaluating LLMs as Human-like Dialogue Systems](https://mlsanity.com/paper/2311.01677/) (2023)
- [MMRC: A Large-Scale Benchmark for Understanding Multimodal Large Language Model in Real-World Conversation](https://mlsanity.com/paper/2502.11903/) (2025)
- [MPCHAT: Towards Multimodal Persona-Grounded Conversation](https://mlsanity.com/paper/2305.17388/) (2023)
- [UniversalRAG: Retrieval-Augmented Generation over Corpora of Diverse Modalities and Granularities](https://mlsanity.com/paper/2504.20734/) (2025)
- [MemEngine: A Unified and Modular Library for Developing Advanced Memory of LLM-based Agents](https://mlsanity.com/paper/2505.02099/) (2025)
- [Agentic Learner with Grow-and-Refine Multimodal Semantic Memory](https://mlsanity.com/paper/2511.21678/) (2025)
- [SABER: Switchable and Balanced Training for Efficient LLM Reasoning](https://mlsanity.com/paper/2508.10026/) (2025)
- [AUGUSTUS: An LLM-Driven Multimodal Agent System with Contextualized User Memory](https://mlsanity.com/paper/2510.15261/) (2025)

## Built on by (most cited first, 30 of 54)

- [A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agents](https://mlsanity.com/paper/2601.12538/) (2026)
- [Code as Agent Harness](https://mlsanity.com/paper/2605.18747/) (2026)
- [A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents](https://mlsanity.com/paper/2602.06052/) (2026)
- [MemoryCD: Benchmarking Long-Context User Memory of LLM Agents for Lifelong Cross-Domain Personalization](https://mlsanity.com/paper/2603.25973/) (2026)
- [Evo-Harness: Context-to-Harness Skill Compilation for Self-Evolving Agents](https://mlsanity.com/paper/2608.15071/) (2026)
- [Omni-SimpleMem: Autoresearch-Guided Discovery of Lifelong Multimodal Agent Memory](https://mlsanity.com/paper/2604.01007/) (2026)
- [MemEye: A Visual-Centric Evaluation Framework for Multimodal Agent Memory](https://mlsanity.com/paper/2605.15128/) (2026)
- [EgoMemReason: A Memory-Driven Reasoning Benchmark for Long-Horizon Egocentric Video Understanding](https://mlsanity.com/paper/2605.09874/) (2026)
- [Prune as You Generate: Online Rollout Pruning for Faster and Better RLVR](https://mlsanity.com/paper/2603.24840/) (2026)
- [Towards Multimodal Lifelong Understanding: A Dataset and Agentic Baseline](https://mlsanity.com/paper/2603.05484/) (2026)
- [MemLens: Benchmarking Multimodal Long-Term Memory in Large Vision-Language Models](https://mlsanity.com/paper/2605.14906/) (2026)
- [AEL: Evolving Agent Harness in Open-Ended Environments](https://mlsanity.com/paper/2604.21725/) (2026)
- [SMMBench: A Benchmark for Source-Distributed Multimodal Agent Memory](https://mlsanity.com/paper/2605.15710/) (2026)
- [MemTrace: Tracing and Attributing Errors in Large Language Model Memory Systems](https://mlsanity.com/paper/2605.28732/) (2026)
- [H2HMem: A Multimodal Memory Benchmark for Agents in Human-Human Interactions](https://mlsanity.com/paper/2606.09461/) (2026)
- [Generalizable Self-Evolving Memory for Automatic Prompt Optimization](https://mlsanity.com/paper/2603.21520/) (2026)
- [M3Exam: Benchmarking Multimodal Memory for Realistic User-Agent Interactions](https://mlsanity.com/paper/2606.07402/) (2026)
- [PaperMind: Benchmarking Agentic Reasoning and Critique over Scientific Papers in Multimodal LLMs](https://mlsanity.com/paper/2604.21304/) (2026)
- [ReContext: Recursive Evidence Replay as LLM Harness for Long-Context Reasoning](https://mlsanity.com/paper/2607.02509/) (2026)
- [Oblivion: Self-Adaptive Agentic Memory Control through Decay-Driven Activation](https://mlsanity.com/paper/2604.00131/) (2026)
- [FileGram: Grounding Agent Personalization in File-System Behavioral Traces](https://mlsanity.com/paper/2604.04901/) (2026)
- [Less Context, More Accuracy: A Bi-Temporal Memory Engine for LLM Agents Where a Lean Retrieved Context Beats the Full History](https://mlsanity.com/paper/2606.09900/) (2026)
- [Mi-Memory: A Lifecycle Memory Framework for Personal AI](https://mlsanity.com/paper/2607.18975/) (2026)
- [Predict, Don't Iterate: Efficient Adaptive-Length Infilling for Diffusion Language Models](https://mlsanity.com/paper/2609.02108/) (2026)
- [The Semantic Lifecycle in Embodied AI: Acquisition, Representation and Storage via Foundation Models](https://mlsanity.com/paper/2601.08876/) (2026)
- [M3Eval: Multi-Modal Memory Evaluation through Cognitively-Grounded Video Tasks](https://mlsanity.com/paper/2606.05008/) (2026)
- [Do Agents Dream of False Memories? Black-box Visual Attacks on Long-term Memory in Multimodal AI Agents](https://mlsanity.com/paper/2607.15657/) (2026)
- [PathMem: Toward Cognition-Aligned Memory Transformation for Pathology MLLMs](https://mlsanity.com/paper/2603.09943/) (2026)
- [Task-Focused Memorization for Multimodal Agents](https://mlsanity.com/paper/2605.31075/) (2026)
- [MemLeak: Diagnosing Information Leaks in Multimodal Agent Memory](https://mlsanity.com/paper/2606.29788/) (2026)

## Abstract (arXiv)

Long-term memory is a critical capability for multimodal large language model (MLLM) agents, particularly in conversational settings where information accumulates and evolves over time. However, existing benchmarks either evaluate multi-session memory in text-only conversations or assess multimodal understanding within localized contexts, failing to evaluate how multimodal memory is preserved, organized, and evolved across long-term conversational trajectories. Thus, we introduce Mem-Gallery, a new benchmark for evaluating multimodal long-term conversational memory in MLLM agents. Mem-Gallery features high-quality multi-session conversations grounded in both visual and textual information, with long interaction horizons and rich multimodal dependencies. Building on this dataset, we propose a systematic evaluation framework that assesses key memory capabilities along three functional dimensions: memory extraction and test-time adaptation, memory reasoning, and memory knowledge management. Extensive benchmarking across thirteen memory systems reveals several key findings, highlighting the necessity of explicit multimodal information retention and memory organization, the persistent limitations in memory reasoning and knowledge management, as well as the efficiency bottleneck of current models.