mlsanityData to 7 Oct 2026

Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Tasks

arXiv 2408.03615 ↗First posted 7 Aug 2024cs.AI cs.CLcited by 83

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Building a general-purpose agent is a long-standing vision in the field of artificial intelligence. Existing agents have made remarkable progress in many domains, yet they still struggle to complete long-horizon tasks in an open world. We attribute this to the lack of necessary world knowledge and multimodal experience that can guide agents through a variety of long-horizon tasks. In this paper, we propose a Hybrid Multimodal Memory module to address the above challenges. It 1) transforms knowledge into Hierarchical Directed Knowledge Graph that allows agents to explicitly represent and learn world knowledge, and 2) summarises historical information into Abstracted Multimodal Experience Pool that provide agents with rich references for in-context learning. On top of the Hybrid Multimodal Memory module, a multimodal agent, Optimus-1, is constructed with dedicated Knowledge-guided Planner and Experience-Driven Reflector, contributing to a better planning and reflection in the face of long-horizon tasks in Minecraft. Extensive experimental results show that Optimus-1 significantly outperforms all existing agents on challenging long-horizon task benchmarks, and exhibits near human-level performance on many tasks. In addition, we introduce various Multimodal Large Language Models (MLLMs) as the backbone of Optimus-1. Experimental results show that Optimus-1 exhibits strong generalization with the help of the Hybrid Multimodal Memory module, outperforming the GPT-4V baseline on many tasks.

Key facts. Cited by 83 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 46 of its 63 references that are in the map. First posted on arXiv 7 Aug 2024.

Builds on

46 of 63 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Models10 Nov
  2. 2023Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memory25 May
  3. 2023Voyager: An Open-Ended Embodied Agent with Large Language Models25 May
  4. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge17 Jun

Other references · 42

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  3. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  4. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 622
  5. 2021Editing Factual Knowledge in Language Modelscited by 548
  6. 2021Fast Model Editing at Scalecited by 491
  7. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  8. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 407
  9. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  10. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 396
  11. 2023An Embodied Generalist Agent in 3D Worldcited by 355
  12. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
30 more
  1. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
  2. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  3. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 223
  4. 2019MineRL: A Large-Scale Dataset of Minecraft Demonstrationscited by 210
  5. 2024LongVLM: Efficient Long Video Understanding via Large Language Modelscited by 146
  6. 2023Detecting and Grounding Multi-Modal Media Manipulationcited by 98
  7. 2017Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learningcited by 97
  8. 2023STEVE-1: A Generative Model for Text-to-Behavior in Minecraftcited by 95
  9. 2024Detoxifying Large Language Models via Knowledge Editingcited by 86
  10. 2024A Survey on Self-Evolution of Large Language Modelscited by 86
  11. 2023LION : Empowering Multimodal Large Language Model with Dual-Level Visual Knowledgecited by 80
  12. 2023MP5: A Multi-modal Open-ended Embodied System in Minecraft via Active Perceptioncited by 67
  13. 2024Scaling Instructable Agents Across Many Simulated Worldscited by 67
  14. 2023Exploring Diverse In-Context Configurations for Image Captioningcited by 63
  15. 2023Detecting and Grounding Multi-Modal Media Manipulation and Beyondcited by 55
  16. 2024CAT: Enhancing Multimodal Large Language Model to Answer Questions in Dynamic Audio-Visual Scenarioscited by 53
  17. 2024MineDreamer: Learning to Follow Instructions via Chain-of-Imagination for Simulated-World Controlcited by 52
  18. 2022EmoCaps: Emotion Capsule based Model for Conversational Emotion Recognitioncited by 48
  19. 2024MoME: Mixture of Multimodal Experts for Generalist Multimodal Large Language Modelscited by 45
  20. 2023GROOT: Learning to Follow Instructions by Watching Gameplay Videoscited by 42
  21. 2023Open-World Multi-Task Control Through Goal-Aware Representation Learning and Adaptive Horizon Predictioncited by 39
  22. 2023Memory Sandbox: Transparent and Interactive Memory Management for Conversational Agentscited by 36
  23. 2023Skill Reinforcement Learning and Planning for Open-World Long-Horizon Taskscited by 35
  24. 2024Enhancing Emotional Generation Capability of Large Language Models via Emotional Chain-of-Thoughtcited by 34
  25. 2023Giraffe: Adventures in Expanding Context Lengths in LLMscited by 32
  26. 2022Generating Executable Action Plans with Environmentally-Aware Language Modelscited by 21
  27. 2023UniSA: Unified Generative Framework for Sentiment Analysiscited by 17
  28. 2024Decision Mamba: A Multi-Grained State Space Model with Self-Evolution Regularization for Offline RLcited by 15
  29. 2024GenView: Enhancing View Quality with Pretrained Generative Model for Self-Supervised Learningcited by 8
  30. 2024Mamba-FSCIL: Dynamic Adaptation with Selective State Space Model for Few-Shot Class-Incremental Learningcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 45; 2026: 31.

83 papers.

Sort:

By area:

  1. 2025Memory in the Age of AI Agentscited by 285
  2. 2025Evaluation and Benchmarking of LLM Agents: A Surveycited by 145
  3. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  4. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 66
  5. 2025LION-FS: Fast & Slow Video-Language Thinker as Online Video Assistantcited by 56
  6. 2025A Survey on the Optimization of Large Language Model-based Agentscited by 45
  7. 2024Kinetix: Investigating the Training of General Agents through Open-Ended Physics-Based Control Taskscited by 40
  8. 2025Rethinking Memory in LLM based Agents: Representations, Operations, and Emerging Topicscited by 36
  9. 2025GUI-explorer: Autonomous Exploration and Mining of Transition-aware Knowledge for GUI Agentcited by 35
  10. 2025Spatial Understanding from Videos: Structured Prompts Meet Simulation Datacited by 28
  11. 2025JARVIS-VLA: Post-Training Large-Scale Vision Language Models to Play Visual Games with Keyboards and Mousecited by 27
  12. 2024WALL-E: World Alignment by Rule Learning Improves World Model-based LLM Agentscited by 24
38 more
  1. 2025Optimus-2: Multimodal Minecraft Agent with Goal-Observation-Action Conditioned Policycited by 24
  2. 2025Embodied AI: From LLMs to World Modelscited by 23
  3. 2025Game-TARS: Pretrained Foundation Models for Scalable Generalist Multimodal Game Agentscited by 23
  4. 2026Graph-based Agent Memory: Taxonomy, Techniques, and Applicationscited by 22
  5. 2024DKDM: Data-Free Knowledge Distillation for Diffusion Models with Any Architecturecited by 21
  6. 2025FALCON: Resolving Visual Redundancy and Fragmentation in High-resolution Multimodal Large Language Models via Visual Registerscited by 20
  7. 2025SpatialScore: Towards Comprehensive Evaluation for Spatial Intelligencecited by 19
  8. 2026Global Prior Meets Local Consistency: Dual-Memory Augmented Vision-Language-Action Model for Efficient Robotic Manipulationcited by 19
  9. 2025Mirage-1: Augmenting and Updating GUI Agent with Hierarchical Multimodal Skillscited by 18
  10. 2025LLM-Powered Decentralized Generative Agents with Adaptive Hierarchical Knowledge Graph for Cooperative Planningcited by 17
  11. 2025Time-IMM: A Dataset and Benchmark for Irregular Multimodal Multivariate Time Seriescited by 16
  12. 2025Coarse-to-Fine Grounded Memory for LLM Agent Planningcited by 16
  13. 2026PersonalAlign: Hierarchical Implicit Intent Alignment for Personalized GUI Agent with Long-Term User-Centric Recordscited by 16
  14. 2025STAR: Learning Diverse Robot Skill Abstractions through Rotation-Augmented Vector Quantizationcited by 13
  15. 2025MEMTRACK: Evaluating Long-Term Memory and State Tracking in Multi-Platform Dynamic Agent Environmentscited by 13
  16. 2025HiconAgent: History Context-aware Policy Optimization for GUI Agentscited by 13
  17. 2026Co-Evolving LLM Decision and Skill Bank Agents for Long-Horizon Taskscited by 13
  18. 2025Optimus-3: Dual-Router Aligned Mixture-of-Experts Agent with Dual-Granularity Reasoning-Aware Policy Optimizationcited by 12
  19. 2024MLLM as Retriever: Interactively Learning Multimodal Retrieval for Embodied Agentscited by 11
  20. 2025PUMA: Layer-Pruned Language Model for Efficient Unified Multimodal Retrieval with Modality-Adaptive Learningcited by 11
  21. 2025Agent Trading Arena: A Study on Numerical Understanding in LLM-Based Agentscited by 9
  22. 2026DeltaVLA: Prior-Guided Vision-Language-Action Models via World Knowledge Variationcited by 9
  23. 2026MementoGUI: Learning Agentic Multimodal Memory Control for Long-Horizon GUI Agentscited by 9
  24. 2024MrSteve: Instruction-Following Agents in Minecraft with What-Where-When Memorycited by 8
  25. 2025When Continue Learning Meets Multimodal Large Language Model: A Surveycited by 8
  26. 2025CausalMACE: Causality Empowered Multi-Agents in Minecraft Cooperative Taskscited by 8
  27. 2025Beyond Pipelines: A Survey of the Paradigm Shift toward Model-Native Agentic AIcited by 8
  28. 2025Open-World Skill Discovery from Unsegmented Demonstrationscited by 7
  29. 2025MemOrb: A Plug-and-Play Verbal-Reinforcement Memory Layer for E-Commerce Customer Servicecited by 7
  30. 2026Evolving Programmatic Skill Networkscited by 7
  31. 2024SELU: Self-Learning Embodied MLLMs in Unknown Environmentscited by 6
  32. 2025EscapeCraft: A 3D Room Escape Environment for Benchmarking Complex Multimodal Reasoning Abilitycited by 6
  33. 2024MindForge: Empowering Embodied Agents with Theory of Mind for Lifelong Cultural Learningcited by 5
  34. 2025Parallelized Planning-Acting for Efficient LLM-based Multi-Agent Systems in Minecraftcited by 5
  35. 2025Uncertainty in Action: Confidence Elicitation in Embodied Agentscited by 5
  36. 2025WALL-E 2.0: World Alignment by NeuroSymbolic Learning improves World Model-based LLM Agentscited by 5
  37. 2025DETACH: Cross-domain Learning for Long-Horizon Tasks via Mixture of Disentangled Expertscited by 5
  38. 2025Agentic Systems in Radiology: Design, Applications, Evaluation, and Challengescited by 5

50 most cited of 83 All 83 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025VistaWise: Building Cost-Effective Agent with Cross-Modal Knowledge Graph for Minecraft
  2. 2025Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with Long-Term Memory
  3. 2024TeamCraft: A Benchmark for Multi-Modal Multi-Agent Systems in Minecraft
  4. 2026MineNPC-Task: Task Suite for Memory-Aware Minecraft Agents
  5. 2023See and Think: Embodied Agent in Virtual Environment
  6. 2025OpenHA: A Series of Open-Source Hierarchical Agentic Models in Minecraft
  7. 2026MineExplorer: Evaluating Open-World Exploration of MLLM Agents in Minecraft
  8. 2025MemVerse: Multimodal Memory for Lifelong Learning Agents