# M3Eval: Multi-Modal Memory Evaluation through Cognitively-Grounded Video Tasks

arXiv: https://arxiv.org/abs/2606.05008 · HTML: https://mlsanity.com/paper/2606.05008/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 91 references that are in the map.
- First posted on arXiv 3 Jun 2026.

## Builds on (most cited first, 30 of 50)

- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [π0.5: a Vision-Language-Action Model with Open-World Generalization](https://mlsanity.com/paper/2504.16054/) (2025)
- [InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency](https://mlsanity.com/paper/2508.18265/) (2025)
- [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://mlsanity.com/paper/2405.21075/) (2024)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](https://mlsanity.com/paper/2311.17005/) (2023)
- [DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence](https://mlsanity.com/paper/2606.19348/) (2026)
- [Evaluating Very Long-Term Conversational Memory of LLM Agents](https://mlsanity.com/paper/2402.17753/) (2024)
- [LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory](https://mlsanity.com/paper/2410.10813/) (2024)
- [A Survey on the Memory Mechanism of Large Language Model based Agents](https://mlsanity.com/paper/2404.13501/) (2024)
- [VAD: Vectorized Scene Representation for Efficient Autonomous Driving](https://mlsanity.com/paper/2303.12077/) (2023)
- [MovieChat: From Dense Token to Sparse Memory for Long Video Understanding](https://mlsanity.com/paper/2307.16449/) (2023)
- [DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models](https://mlsanity.com/paper/2402.12289/) (2024)
- [LVBench: An Extreme Long Video Understanding Benchmark](https://mlsanity.com/paper/2406.08035/) (2024)
- [Memory in the Age of AI Agents](https://mlsanity.com/paper/2512.13564/) (2025)
- [MLVU: Benchmarking Multi-task Long Video Understanding](https://mlsanity.com/paper/2406.04264/) (2024)
- [VideoAgent: A Memory-augmented Multimodal Agent for Video Understanding](https://mlsanity.com/paper/2403.11481/) (2024)
- [BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack](https://mlsanity.com/paper/2406.10149/) (2024)
- [StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding](https://mlsanity.com/paper/2411.03628/) (2024)
- [OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding?](https://mlsanity.com/paper/2501.05510/) (2025)
- [ChatDB: Augmenting LLMs with Databases as Their Symbolic Memory](https://mlsanity.com/paper/2306.03901/) (2023)
- [Evo-Memory: Benchmarking LLM Agent Test-time Learning with Self-Evolving Memory](https://mlsanity.com/paper/2511.20857/) (2025)
- [Cambrian-S: Towards Spatial Supersensing in Video](https://mlsanity.com/paper/2511.04670/) (2025)
- [Video-Holmes: Can MLLM Think Like Holmes for Complex Video Reasoning?](https://mlsanity.com/paper/2505.21374/) (2025)
- [HourVideo: 1-Hour Video-Language Understanding](https://mlsanity.com/paper/2411.04998/) (2024)
- [Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with Long-Term Memory](https://mlsanity.com/paper/2508.09736/) (2025)
- [EgoLife: Towards Egocentric Life Assistant](https://mlsanity.com/paper/2503.03803/) (2025)
- [MemBench: Towards More Comprehensive Evaluation on the Memory of LLM-based Agents](https://mlsanity.com/paper/2506.21605/) (2025)
- [ReconVLA: Reconstructive Vision-Language-Action Model as Effective Robot Perceiver](https://mlsanity.com/paper/2508.10333/) (2025)
- [LifelongAgentBench: Evaluating LLM Agents as Lifelong Learners](https://mlsanity.com/paper/2505.11942/) (2025)

## Built on by (most cited first, 2 of 2)

- [Autonomous Research Agents: A Survey of AI Scientists and the Verification Gap](https://mlsanity.com/paper/2608.05179/) (2026)
- [EmbodiedCLBench: Evaluating Continual Learning for Self-Evolving Embodied Agents](https://mlsanity.com/paper/2610.13021/) (2026)

## Abstract (arXiv)

As multi-modal models advance towards long-form video understanding, memory emerges as a critical capability. Despite substantial efforts in developing video datasets and benchmarks, existing works primarily focus on perception and reasoning, without systematically evaluating memory: what models retain, how faithfully information is preserved, and how robust memory remains under interference. To address this gap, we introduce M3Eval, the first comprehensive evaluation framework and benchmark for probing different memory dimensions in multi-modal models. Grounded in cognitive psychology, our design features carefully constructed tasks that isolate key aspects of memory. Leveraging M3Eval, we conduct extensive experiments across representative multi-modal models, revealing consistent weaknesses and distinctive behaviors. We find that models struggle to maintain disentangled representations when processing parallel video streams, exhibit interference patterns differing substantially from those observed in human memory, ground memory sources more reliably in the spatial domain than the temporal domain, and demonstrate limited symbolic memory. Collectively, our benchmark provides a valuable resource for future research, while our findings highlight memory as a fundamental yet underexplored capability and offer insights for designing more effective memory mechanisms in multi-modal models. Our code and dataset are available at https://pku-value-lab.github.io/m3eval-homepage.