mlsanityData to 7 Oct 2026

Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with Long-Term Memory

arXiv 2508.09736 ↗First posted 13 Aug 2025cs.CVcited by 88

On the map: LLM agents and tool use → Multimodal and embodied agents

Abstract (arXiv)

We introduce M3-Agent, a novel multimodal agent framework equipped with long-term memory. Like humans, M3-Agent can process real-time visual and auditory inputs to build and update episodic and semantic memories, gradually accumulating world knowledge. Its memory is organized in an entity-centric, multimodal manner, enabling deeper and more consistent understanding of the environment. Given an instruction, M3-Agent autonomously performs multi-turn reasoning and retrieves relevant memories to complete tasks. To evaluate memory effectiveness and memory-based reasoning in multimodal agents, we develop M3-Bench, a long-video question answering benchmark comprising 100 newly recorded robot-perspective videos (M3-Bench-robot) and 920 diverse web-sourced videos (M3-Bench-web). We annotate QA pairs designed to test capabilities essential for agent applications, such as person understanding, general knowledge extraction, and cross-modal reasoning. Experimental results show that M3-Agent, trained via reinforcement learning, outperforms the strongest baseline, a prompting agent using Gemini-1.5-pro and GPT-4o, achieving 6.7%, 7.7%, and 5.3% higher accuracy on M3-Bench-robot, M3-Bench-web and VideoMME-long, respectively. Our work advances multimodal agents toward more human-like long-term memory and provides insights for their practical design. Model, code and data are available at https://github.com/bytedance-seed/m3-agent.

Key facts. Cited by 88 papers in the map (data through 7 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 60 of its 62 references that are in the map. First posted on arXiv 13 Aug 2025.

Builds on

60 of 62 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024VideoAgent: A Memory-augmented Multimodal Agent for Video Understanding18 Mar
  2. 2023Open-Ended Instructable Embodied Agents with Memory-Augmented Large Language Models23 Oct

Other references · 58

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2025Qwen2.5-VL Technical Reportcited by 5,418
  5. 2024GPT-4o System Cardcited by 4,467
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  7. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  8. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  9. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  10. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  11. 2025A-MEM: Agentic Memory for LLM Agentscited by 1,000
  12. 2023VideoChat: Chat-Centric Video Understandingcited by 851
38 more
  1. 2025Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memorycited by 722
  2. 2025Qwen2.5-Omni Technical Reportcited by 704
  3. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  4. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  5. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 644
  6. 2023MemoryBank: Enhancing Large Language Models with Long-Term Memorycited by 539
  7. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  8. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  9. 2024Long Context Transfer from Language to Visioncited by 502
  10. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understandingcited by 341
  11. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 303
  12. 2024MLVU: Benchmarking Multi-task Long Video Understandingcited by 275
  13. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 223
  14. 2024Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understandingcited by 210
  15. 2022MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognitioncited by 189
  16. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understandingcited by 180
  17. 2023JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Modelscited by 170
  18. 2024Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streamscited by 160
  19. 2024AgentSquare: Automatic LLM Agent Search in Modular Design Spacecited by 141
  20. 2024Tarsier: Recipes for Training and Evaluating Large Video Description Modelscited by 140
  21. 2025Memory OS of AI Agentcited by 140
  22. 2025OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding?cited by 134
  23. 2024Hello Again! LLM-powered Personalized Agent for Long-term Dialoguecited by 121
  24. 2024HourVideo: 1-Hour Video-Language Understandingcited by 110
  25. 2024HiAgent: Hierarchical Working Memory Management for Solving Long-Horizon Agent Tasks with Large Language Modelcited by 101
  26. 2023SCM: Enhancing Large Language Model with Self-Controlled Memory Frameworkcited by 98
  27. 2024AIOS: LLM Agent Operating Systemcited by 94
  28. 2024Video ReCap: Recursive Captioning of Hour-Long Videoscited by 88
  29. 2023MM-VID: Advancing Video Understanding with GPT-4V(ision)cited by 82
  30. 2025Tarsier2: Advancing Large Vision-Language Models from Detailed Video Description to Comprehensive Video Understandingcited by 80
  31. 2024Agent Planning with World Knowledge Modelcited by 66
  32. 2025Embodied AI Agents: Modeling the Worldcited by 57
  33. 2024Elements of World Knowledge (EWoK): A Cognition-Inspired Framework for Evaluating Basic World Knowledge in Language Modelscited by 55
  34. 2024AgentLite: A Lightweight Library for Building and Advancing Task-Oriented LLM Agent Systemcited by 53
  35. 2023AutoAD: Movie Description in Contextcited by 50
  36. 2024InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactionscited by 48
  37. 2023MM-Narrator: Narrating Long-form Videos with Multimodal In-Context Learningcited by 46
  38. 2024AGILE: A Novel Reinforcement Learning Framework of LLM Agentscited by 45

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 12; 2026: 76.

88 papers.

Sort:

By area:

  1. 2025Memory in the Age of AI Agentscited by 285
  2. 2025WorldMM: Dynamic Multimodal Memory Agent for Long Video Reasoningcited by 50
  3. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
  4. 2026A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agentscited by 40
  5. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecyclecited by 34
  6. 2025LongVideoAgent: Multi-Agent Reasoning with Long Videoscited by 23
  7. 2026Graph-based Agent Memory: Taxonomy, Techniques, and Applicationscited by 22
  8. 2026OmniGAIA: Towards Native Omni-Modal AI Agentscited by 21
  9. 2025AI for Service: Proactive Assistance with AI Glassescited by 18
  10. 2026GEMS: Agent-Native Multimodal Generation with Memory and Skillscited by 16
  11. 2025TeleMem: Building Long-Term and Multimodal Memory for Agentic AIcited by 14
  12. 2026From Verbatim to Gist: Distilling Pyramidal Multimodal Memory via Semantic Information Bottleneck for Long-Horizon Video Agentscited by 14
38 more
  1. 2026LensWalk: Agentic Video Understanding by Planning How You See in Videoscited by 14
  2. 2025AI Meets Brain: Memory Systems from Cognitive Neuroscience to Autonomous Agentscited by 12
  3. 2026Memex(RL): Scaling Long-Horizon LLM Agents via Indexed Experience Memorycited by 11
  4. 2026EgoMemReason: A Memory-Driven Reasoning Benchmark for Long-Horizon Egocentric Video Understandingcited by 11
  5. 2026EventMemAgent: Hierarchical Event-Centric Memory for Online Video Understanding with Adaptive Tool Usecited by 10
  6. 2026Toward Personalized LLM-Powered Agents: Foundations, Evaluation, and Future Directionscited by 10
  7. 2026PersonaVLM: Long-Term Personalized Multimodal LLMscited by 10
  8. 2025TeleEgo: Benchmarking Egocentric AI Assistants in the Wildcited by 9
  9. 2026Towards Multimodal Lifelong Understanding: A Dataset and Agentic Baselinecited by 8
  10. 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8
  11. 2025ProAgent: Harnessing On-Demand Sensory Contexts for Proactive LLM Agent Systems in the Wildcited by 7
  12. 2026Think While Watching: Online Streaming Segment-Level Memory for Multi-Turn Video Reasoning in Multimodal Large Language Modelscited by 7
  13. 2025MARC: Memory-Augmented RL Token Compression for Efficient Video Understandingcited by 6
  14. 2025Agentic Video Intelligence: A Flexible Framework for Advanced Video Exploration and Understandingcited by 6
  15. 2026MemDreamer: Decoupling Perception and Reasoning for Long Video Understanding via Hierarchical Graph Memory and Agentic Retrieval Mechanismcited by 6
  16. 2026OmniRAG-Agent: Agentic Omnimodal Reasoning for Low-Resource Long Audio-Video Question Answeringcited by 5
  17. 2026Advancing Multimodal Agent Reasoning with Long-Term Neuro-Symbolic Memorycited by 5
  18. 2026EVA: Efficient Reinforcement Learning for End-to-End Video Agentcited by 5
  19. 2025video-SALMONN S: Memory-Enhanced Streaming Audio-Visual LLMcited by 4
  20. 2026StreamingClaw Technical Reportcited by 4
  21. 2026Scaling Video Understanding via Compact Latent Multi-Agent Collaborationcited by 4
  22. 2026From Atom to Community: Structured and Evolving Agent Memory for User Behavior Modelingcited by 3
  23. 2026SpeechLess: Micro-utterance with Personalized Spatial Memory-aware Assistant in Everyday Augmented Realitycited by 3
  24. 2026M2: Dual-Memory Augmentation for Long-Horizon Web Agents via Trajectory Summarization and Insight Retrievalcited by 3
  25. 2026FocusGraph: Graph-Structured Frame Selection for Embodied Long Video Question Answeringcited by 3
  26. 2026FileGram: Grounding Agent Personalization in File-System Behavioral Tracescited by 3
  27. 2026Persistent Visual Memory: Sustaining Perception for Deep Generation in LVLMscited by 3
  28. 2026Bridging Modalities, Spanning Time: Structured Memory for Ultra-Long Agentic Video Reasoningcited by 3
  29. 2026MemLens: Benchmarking Multimodal Long-Term Memory in Large Vision-Language Modelscited by 3
  30. 2026WorldMemArena: Evaluating Multimodal Agent Memory Through Action-World Interactioncited by 3
  31. 2026H2HMem: A Multimodal Memory Benchmark for Agents in Human-Human Interactionscited by 3
  32. 2026Native Active Perception as Reasoning for Omni-Modal Understandingcited by 3
  33. 2026MEMORA: Embodied Action Memory from Egocentric Videos for Reasoning and Planningcited by 3
  34. 2026StreamArena: Toward Continuous, Interactive, and Long-Horizon Agentic Streaming Video Understandingcited by 3
  35. 2026Neuromem: A Granular Decomposition of the Streaming Lifecycle in External Memory for LLMscited by 2
  36. 2026Vinci2: Providing Proactive Assistance in Continuous Egocentric Videoscited by 2
  37. 2026From Memory to Skills: Evidence-Grounded Co-Evolution Governance for Long-Horizon LLM Agentscited by 2
  38. 2026MetaVideoAgent: Automated Video-Agent Evolution for Long-Form Video Understandingcited by 2

50 most cited of 88 All 88 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025MemVerse: Multimodal Memory for Lifelong Learning Agents
  2. 2026PMMC: Prospective Multimodal Memory Compilation for Long-Term LVLM Agents
  3. 2026MemEye: A Visual-Centric Evaluation Framework for Multimodal Agent Memory
  4. 2026InternVideo3: Agentify Foundation Models with Multimodal Contextual Reasoning
  5. 2025Agent-ScanKit: Unraveling Memory and Reasoning of Multimodal Agents via Sensitivity Perturbations
  6. 2025Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions
  7. 2026Memory in the LLM Era: Modular Architectures and Strategies in a Unified Framework
  8. 2026Light-Omni: Reflex over Reasoning in Agentic Video Understanding with Long-Term Memory