mlsanityData to 7 Oct 2026

MP5: A Multi-modal Open-ended Embodied System in Minecraft via Active Perception

arXiv 2312.07472 ↗First posted 12 Dec 2023cs.CVcited by 67

Not on the map. Why?

Cited by 34 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

It is a long-lasting goal to design an embodied system that can solve long-horizon open-world tasks in human-like ways. However, existing approaches usually struggle with compound difficulties caused by the logic-aware decomposition and context-aware execution of these tasks. To this end, we introduce MP5, an open-ended multimodal embodied system built upon the challenging Minecraft simulator, which can decompose feasible sub-objectives, design sophisticated situation-aware plans, and perform embodied action control, with frequent communication with a goal-conditioned active perception scheme. Specifically, MP5 is developed on top of recent advances in Multimodal Large Language Models (MLLMs), and the system is modulated into functional modules that can be scheduled and collaborated to ultimately solve pre-defined context- and process-dependent tasks. Extensive experiments prove that MP5 can achieve a 22% success rate on difficult process-dependent tasks and a 91% success rate on tasks that heavily depend on the context. Moreover, MP5 exhibits a remarkable ability to address many open-ended tasks that are entirely novel.

Key facts. Cited by 67 papers in the map (data through 7 Oct 2026). Builds on 35 of its 45 references that are in the map. First posted on arXiv 12 Dec 2023.

Builds on

35 of 45 references.

References · 35

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  9. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  10. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  11. 2023Visual Instruction Tuningcited by 7,792
  12. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
23 more
  1. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  2. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  3. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  4. 2023Mastering Diverse Domains through World Modelscited by 1,193
  5. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  6. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  7. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  8. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  9. 2022A Generalist Agentcited by 802
  10. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  11. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  12. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 396
  13. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
  14. 2023Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memorycited by 288
  15. 2019MineRL: A Large-Scale Dataset of Minecraft Demonstrationscited by 210
  16. 2023SmartEdit: Exploring Complex Instruction-based Image Editing with Multimodal Large Language Modelscited by 174
  17. 2023JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Modelscited by 170
  18. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  19. 2017Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learningcited by 97
  20. 2023STEVE-1: A Generative Model for Text-to-Behavior in Minecraftcited by 95
  21. 2023Open-World Multi-Task Control Through Goal-Aware Representation Learning and Adaptive Horizon Predictioncited by 39
  22. 2021JueWu-MC: Playing Minecraft with Sample-efficient Hierarchical Reinforcement Learningcited by 28
  23. 2023Reinforcement Learning Friendly Vision-Language Model for Minecraftcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 22; 2025: 37; 2026: 8.

67 papers.

Sort:

By area:

  1. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 867
  2. 2025GameFactory: Creating New Games with Generative Interactive Videoscited by 124
  3. 2024MMedAgent: Learning to Use Medical Tools with Multi-modal Agentcited by 108
  4. 2024Large Multimodal Agents: A Surveycited by 99
  5. 2024Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Taskscited by 83
  6. 2024SPA-VL: A Comprehensive Safety Preference Alignment Dataset for Vision Language Modelcited by 82
  7. 2024PsySafe: A Comprehensive Framework for Psychological-based Attack, Defense, and Evaluation of Multi-agent System Safetycited by 79
  8. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  9. 2024AppAgent v2: Advanced Agent for Flexible Mobile Interactionscited by 67
  10. 2025A Survey on (M)LLM-Based GUI Agentscited by 61
  11. 2024Code-as-Monitor: Constraint-aware Visual Programming for Reactive and Proactive Robotic Failure Detectioncited by 59
  12. 2024MineDreamer: Learning to Follow Instructions via Chain-of-Imagination for Simulated-World Controlcited by 52
38 more
  1. 2025Manipulating Multimodal Agents via Cross-Modal Prompt Injectioncited by 42
  2. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  3. 2024OmniJARVIS: Unified Vision-Language-Action Tokenization Enables Open-World Instruction Following Agentscited by 30
  4. 2025TIGeR: Tool-Integrated Geometric Reasoning in Vision-Language Models for Roboticscited by 27
  5. 2024Odyssey: Empowering Minecraft Agents with Open-World Skillscited by 26
  6. 2025Optimus-2: Multimodal Minecraft Agent with Goal-Observation-Action Conditioned Policycited by 24
  7. 2025Embodied AI: From LLMs to World Modelscited by 23
  8. 2024RH20T-P: A Primitive-Level Robotic Dataset Towards Composable Generalization Agentscited by 20
  9. 2024ROCKET-1: Mastering Open-World Interaction with Visual-Temporal Context Promptingcited by 20
  10. 2025VIKI-R: Coordinating Embodied Multi-Agent Cooperation via Reinforcement Learningcited by 18
  11. 2024MineLand: Simulating Large-Scale Multi-Agent Interactions with Limited Multimodal Senses and Physical Needscited by 16
  12. 2025DreamNav: A Trajectory-Based Imaginative Framework for Zero-Shot Vision-and-Language Navigationcited by 15
  13. 2024ADAM: An Embodied Causal Agent in Open-World Environmentscited by 13
  14. 2025Generative Multi-Agent Collaboration in Embodied AI: A Systematic Reviewcited by 13
  15. 2025Position: Interactive Generative Video as Next-Generation Game Enginecited by 13
  16. 2025VistaWise: Building Cost-Effective Agent with Cross-Modal Knowledge Graph for Minecraftcited by 13
  17. 2025Optimus-3: Dual-Router Aligned Mixture-of-Experts Agent with Dual-Granularity Reasoning-Aware Policy Optimizationcited by 12
  18. 2025NavigateDiff: Visual Predictors are Zero-Shot Navigation Assistantscited by 11
  19. 2025A Survey of Large Language Model-Powered Spatial Intelligence Across Scales: Advances in Embodied Agents, Smart Cities, and Earth Sciencecited by 11
  20. 2025Prune4Web: DOM Tree Pruning Programming for Web Agentcited by 11
  21. 2025RoboSafe: Safeguarding Embodied Agents via Executable Safety Logiccited by 11
  22. 2025MAPS: Advancing Multi-Modal Reasoning in Expert-Level Physical Sciencecited by 10
  23. 2025MapAgent: Trajectory-Constructed Memory-Augmented Planning for Mobile Task Automationcited by 10
  24. 2024AGFSync: Leveraging AI-Generated Feedback for Preference Optimization in Text-to-Image Generationcited by 9
  25. 2024Story3D-Agent: Exploring 3D Storytelling Visualization with Large Language Modelscited by 9
  26. 2024MrSteve: Instruction-Following Agents in Minecraft with What-Where-When Memorycited by 8
  27. 2025FOUNDER: Grounding Foundation Models in World Models for Open-Ended Embodied Decision Makingcited by 8
  28. 2024Exploring the Potential of Large Language Models for Improving Digital Forensic Investigation Efficiencycited by 7
  29. 2025Open-World Skill Discovery from Unsegmented Demonstrationscited by 7
  30. 2025Towards Robust Evaluation of STEM Education: Leveraging MLLMs in Project-Based Learningcited by 7
  31. 2025DVM: Towards Controllable LLM Agents in Social Deduction Gamescited by 6
  32. 2025EscapeCraft: A 3D Room Escape Environment for Benchmarking Complex Multimodal Reasoning Abilitycited by 6
  33. 2026SaPaVe: Towards Active Perception and Manipulation in Vision-Language-Action Models for Roboticscited by 6
  34. 2025Parallelized Planning-Acting for Efficient LLM-based Multi-Agent Systems in Minecraftcited by 5
  35. 2025Uncertainty in Action: Confidence Elicitation in Embodied Agentscited by 5
  36. 2025WALL-E 2.0: World Alignment by NeuroSymbolic Learning improves World Model-based LLM Agentscited by 5
  37. 2025PhysiAgent: An Embodied Agent Framework in Physical Worldcited by 5
  38. 2026Experience Transfer for Multimodal LLM Agents in Minecraft Gamecited by 4

50 most cited of 67 All 67 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026MineExplorer: Evaluating Open-World Exploration of MLLM Agents in Minecraft
  2. 2023See and Think: Embodied Agent in Virtual Environment
  3. 2025OpenHA: A Series of Open-Source Hierarchical Agentic Models in Minecraft
  4. 2024TeamCraft: A Benchmark for Multi-Modal Multi-Agent Systems in Minecraft
  5. 2025Collaborating Action by Action: A Multi-agent LLM Framework for Embodied Reasoning
  6. 2026CEDAR: Automata as Verifiable Interfaces for Language-Guided Embodied Action
  7. 2025P3: Toward Versatile Embodied Agents
  8. 2026ABot-Claw: A Foundation for Persistent, Cooperative, and Self-Evolving Robotic Agents