mlsanityData to 14 Oct 2026

Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memory

arXiv 2305.17144 ↗First posted 25 May 2023cs.AI cs.CL cs.CVcited by 289

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

The captivating realm of Minecraft has attracted substantial research interest in recent years, serving as a rich platform for developing intelligent agents capable of functioning in open-world environments. However, the current research landscape predominantly focuses on specific objectives, such as the popular "ObtainDiamond" task, and has not yet shown effective generalization to a broader spectrum of tasks. Furthermore, the current leading success rate for the "ObtainDiamond" task stands at around 20%, highlighting the limitations of Reinforcement Learning (RL) based controllers used in existing methods. To tackle these challenges, we introduce Ghost in the Minecraft (GITM), a novel framework integrates Large Language Models (LLMs) with text-based knowledge and memory, aiming to create Generally Capable Agents (GCAs) in Minecraft. These agents, equipped with the logic and common sense capabilities of LLMs, can skillfully navigate complex, sparse-reward environments with text-based interactions. We develop a set of structured actions and leverage LLMs to generate action plans for the agents to execute. The resulting LLM-based agent markedly surpasses previous methods, achieving a remarkable improvement of +47.5% in success rate on the "ObtainDiamond" task, demonstrating superior robustness compared to traditional RL-based controllers. Notably, our agent is the first to procure all items in the Minecraft Overworld technology tree, demonstrating its extensive capabilities. GITM does not need any GPU for training, but a single CPU node with 32 CPU cores is enough. This research shows the potential of LLMs in developing capable agents for handling long-horizon, complex tasks and adapting to uncertainties in open-world environments. See the project website at https://github.com/OpenGVLab/GITM.

Key facts. Cited by 289 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 22 of its 28 references that are in the map. First posted on arXiv 25 May 2023.

Builds on

22 of 28 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge17 Jun

Other references · 21

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  3. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  4. 2023Mastering Diverse Domains through World Modelscited by 1,253
  5. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
  6. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  7. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  8. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  9. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 611
  10. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 398
  11. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 345
  12. 2021Open-Ended Learning Leads to Generally Capable Agentscited by 182
9 more
  1. 2016A Deep Hierarchical Approach to Lifelong Learning in Minecraftcited by 90
  2. 2023Open-World Multi-Task Control Through Goal-Aware Representation Learning and Adaptive Horizon Predictioncited by 39
  3. 2021Multi-task curriculum learning in a complex, visual, hard-exploration domain: Minecraftcited by 35
  4. 2021JueWu-MC: Playing Minecraft with Sample-efficient Hierarchical Reinforcement Learningcited by 28
  5. 2022MineRL Diamond 2021 Competition: Overview, Results, and Lessons Learnedcited by 26
  6. 2021SEIHAI: A Sample-efficient Hierarchical AI for the MineRL Competitioncited by 20
  7. 2020Retrospective Analysis of the 2019 MineRL Competition on Sample Efficient Reinforcement Learningcited by 18
  8. 2023Towards Solving Fuzzy Tasks with Human Feedback: A Retrospective of the MineRL BASALT 2022 Competitioncited by 15
  9. 2020Scaling Imitation Learning in Minecraftcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 59; 2024: 110; 2025: 70; 2026: 50.

289 papers.

Sort:

By area:

  1. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  2. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  3. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  4. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  5. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 1,011
  6. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 634
  7. 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Modelscited by 574
  8. 2024Executable Code Actions Elicit Better LLM Agentscited by 564
  9. 2023A Dynamic LLM-Powered Agent Network for Task-Oriented Agent Collaborationcited by 290
  10. 2023Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectivescited by 257
  11. 2023MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Usecited by 217
  12. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 212
38 more
  1. 2023DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large Language Modelscited by 207
  2. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 205
  3. 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 194
  4. 2023LanguageMPC: Large Language Models as Decision Makers for Autonomous Drivingcited by 190
  5. 2025AgentSociety: Large-Scale Simulation of LLM-Driven Generative Agents Advances Understanding of Human Behaviors and Societycited by 186
  6. 2023You Only Look at Screens: Multimodal Chain-of-Action Agentscited by 185
  7. 2023JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Modelscited by 171
  8. 2024Large Language Model-Brained GUI Agents: A Surveycited by 162
  9. 2023Drive Like a Human: Rethinking Autonomous Driving with Large Language Modelscited by 160
  10. 2024AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?cited by 158
  11. 2023Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Gamecited by 152
  12. 2023User Behavior Simulation with Large Language Model based Agentscited by 140
  13. 2023Large Language Models for Robotics: A Surveycited by 133
  14. 2024Exploring Large Language Model based Intelligent Agents: Definitions, Methods, and Prospectscited by 132
  15. 2025A Survey of Context Engineering for Large Language Modelscited by 126
  16. 2025G-Memory: Tracing Hierarchical Memory for Multi-Agent Systemscited by 122
  17. 2024Empowering Biomedical Discovery with AI Agentscited by 118
  18. 2023The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open Worldcited by 116
  19. 2024FinCon: A Synthesized LLM Multi-Agent System with Conceptual Verbal Reinforcement for Enhanced Financial Decision Makingcited by 113
  20. 2024A Survey on Large Language Model-Based Game Agentscited by 112
  21. 2023Skeleton-of-Thought: Prompting LLMs for Efficient Parallel Generationcited by 105
  22. 2023Avalon's Game of Thoughts: Battle Against Deception through Recursive Contemplationcited by 104
  23. 2024Large Language Models Meet NLP: A Surveycited by 103
  24. 2024VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agentscited by 103
  25. 2024AriGraph: Learning Knowledge Graph World Models with Episodic Memory for LLM Agentscited by 102
  26. 2023Drive as You Speak: Enabling Human-Like Interaction with Large Language Models in Autonomous Vehiclescited by 100
  27. 2024A Review of Large Language Models and Autonomous Agents in Chemistrycited by 99
  28. 2024AIOS: LLM Agent Operating Systemcited by 97
  29. 2024Small LLMs Are Weak Tool Learners: A Multi-LLM Agentcited by 93
  30. 2025AGrail: A Lifelong Agent Guardrail with Effective and Adaptive Safety Detectioncited by 89
  31. 2025MemInsight: Autonomous Memory Augmentation for LLM Agentscited by 85
  32. 2024Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Taskscited by 84
  33. 2024AgentGen: Enhancing Planning Abilities for Large Language Model based Agent via Environment and Task Generationcited by 83
  34. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 83
  35. 2024Embodied LLM Agents Learn to Cooperate in Organized Teamscited by 82
  36. 2024From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agentscited by 82
  37. 2023Large Language Models Play StarCraft II: Benchmarks and A Chain of Summarization Approachcited by 79
  38. 2024If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agentscited by 79

50 most cited of 289 All 289 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Do Embodied Agents Dream of Pixelated Sheep: Embodied Decision Making using Language Guided World Modelling
  2. 2026MineExplorer: Evaluating Open-World Exploration of MLLM Agents in Minecraft
  3. 2025OpenHA: A Series of Open-Source Hierarchical Agentic Models in Minecraft
  4. 2025Training Agents Inside of Scalable World Models
  5. 2024STEVE Series: Step-by-Step Construction of Agent Systems in Minecraft
  6. 2024LARM: Large Auto-Regressive Model for Long-Horizon Embodied Intelligence
  7. 2023Reinforcement Learning Friendly Vision-Language Model for Minecraft
  8. 2025Optimus-2: Multimodal Minecraft Agent with Goal-Observation-Action Conditioned Policy