mlsanityData to 7 Oct 2026

MCU: An Evaluation Framework for Open-Ended Game Agents

arXiv 2310.08367 ↗First posted 12 Oct 2023cs.AI cs.CL cs.CVcited by 33

Not on the map. Why?

Abstract (arXiv)

Developing AI agents capable of interacting with open-world environments to solve diverse tasks is a compelling challenge. However, evaluating such open-ended agents remains difficult, with current benchmarks facing scalability limitations. To address this, we introduce Minecraft Universe (MCU), a comprehensive evaluation framework set within the open-world video game Minecraft. MCU incorporates three key components: (1) an expanding collection of 3,452 composable atomic tasks that encompasses 11 major categories and 41 subcategories of challenges; (2) a task composition mechanism capable of generating infinite diverse tasks with varying difficulty; and (3) a general evaluation framework that achieves 91.5% alignment with human ratings for open-ended task assessment. Empirical results reveal that even state-of-the-art foundation agents struggle with the increasing diversity and complexity of tasks. These findings highlight the necessity of MCU as a robust benchmark to drive progress in AI agent development within open-ended environments. Our evaluation code and scripts are available at https://github.com/CraftJarvis/MCU.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Builds on 37 of its 90 references that are in the map. First posted on arXiv 12 Oct 2023.

Builds on

37 of 90 references.

References · 37

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  6. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  7. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  8. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  9. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  10. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  11. 2022A Generalist Agentcited by 802
  12. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
25 more
  1. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  2. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 396
  3. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
  4. 2023Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memorycited by 288
  5. 2019MineRL: A Large-Scale Dataset of Minecraft Demonstrationscited by 210
  6. 2017Teacher-Student Curriculum Learningcited by 197
  7. 2021Benchmarking the Spectrum of Agent Capabilitiescited by 193
  8. 2021Open-Ended Learning Leads to Generally Capable Agentscited by 177
  9. 2023Benchmarking Foundation Models with Language-Model-as-an-Examinercited by 172
  10. 2023GenSim: Generating Robotic Simulation Tasks via Large Language Modelscited by 120
  11. 2023PRD: Peer Rank and Discussion Improve Large Language Model based Evaluationscited by 114
  12. 2023STEVE-1: A Generative Model for Text-to-Behavior in Minecraftcited by 95
  13. 2016A Deep Hierarchical Approach to Lifelong Learning in Minecraftcited by 90
  14. 2017Hierarchical and Interpretable Skill Acquisition in Multi-task Reinforcement Learningcited by 77
  15. 2019Keeping Your Distance: Solving Sparse Reward Tasks Using Self-Balancing Shaped Rewardscited by 76
  16. 2016Control of Memory, Active Perception, and Action in Minecraftcited by 61
  17. 2019Deep Reinforcement Learning from Policy-Dependent Human Feedbackcited by 56
  18. 2021In-IDE Code Generation from Natural Language: Promise and Challengescited by 41
  19. 2021Open-world Machine Learning: Applications, Challenges, and Opportunitiescited by 34
  20. 2019The MineRL 2019 Competition on Sample Efficient Reinforcement Learning using Human Priorscited by 25
  21. 2021The MineRL BASALT Competition on Learning from Human Feedbackcited by 24
  22. 2023Mini-BEHAVIOR: A Procedurally Generated Benchmark for Long-horizon Decision-Making in Embodied AIcited by 23
  23. 2023Reinforcement Learning Friendly Vision-Language Model for Minecraftcited by 14
  24. 2018Deep Reinforcement Learning with Model Learning and Monte Carlo Tree Search in Minecraftcited by 5
  25. 2021High precision control and deep learning-based corn stand counting algorithms for agricultural robotcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 7; 2025: 8; 2026: 17.

33 papers.

Sort:

By area:

  1. 2025A Survey on Vision-Language-Action Models: An Action Tokenization Perspectivecited by 96
  2. 2024RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Horizon Generationcited by 75
  3. 2024VillagerAgent: A Graph-Based Multi-Agent Framework for Coordinating Complex Task Dependencies in Minecraftcited by 34
  4. 2023Creative Agents: Empowering Agents with Imagination for Creative Taskscited by 24
  5. 2024VideoLLaMB: Long Streaming Video Understanding with Recurrent Memory Bridgescited by 20
  6. 2025SoMi-ToM: Evaluating Multi-Perspective Theory of Mind in Embodied Social Interactionscited by 16
  7. 2024Computational Experiments Meet Large Language Model Based Agents: A Survey and Perspectivecited by 11
  8. 2024Mars: Situated Inductive Reasoning in an Open-World Environmentcited by 10
  9. 2024GROOT-2: Weakly Supervised Multi-Modal Instruction Following Agentscited by 9
  10. 2025FinGAIA: A Chinese Benchmark for AI Agents in Real-World Financial Domaincited by 8
  11. 2026Risky-Bench: Probing Agentic Safety Risks under Real-World Deploymentcited by 8
  12. 2024Luban: Building Open-Ended Creative Agents via Autonomous Embodied Verificationcited by 5
21 more
  1. 2025AVA: Attentive VLM Agent for Mastering StarCraft IIcited by 5
  2. 2025EmbRACE-3K: Embodied Reasoning and Action in Complex Environmentscited by 5
  3. 2026Experience Transfer for Multimodal LLM Agents in Minecraft Gamecited by 4
  4. 2025GraphicBench: A Planning Benchmark for Graphic Design with Language Agentscited by 3
  5. 2025TongSIM: A General Platform for Simulating Intelligent Machinescited by 3
  6. 2026MineExplorer: Evaluating Open-World Exploration of MLLM Agents in Minecraftcited by 3
  7. 2026MAIN-VLA: Modeling Abstraction of Intention and eNvironment for Vision-Language-Action Modelscited by 2
  8. 2026Towards Generalist Game Players: An Investigation of Foundation Models in the Game Multiversecited by 2
  9. 2025Can Agent Conquer Web? Exploring the Frontiers of ChatGPT Atlas Agent in Web Gamescited by 1
  10. 2026Automatic Cognitive Task Generation for In-Situ Evaluation of Embodied Agentscited by 1
  11. 2026Mastering PokeGym: Graph-Guided Multimodal Evolution at Test Timecited by 1
  12. 2026Can Current Agents Close the Discovery-to-Application Gap? A Case Study in Minecraftcited by 1
  13. 2026Multi-agent Framework for Time-Sensitive Complementary Collaboration in Minecraftcited by 1
  14. 2026MirrorCraft: Paired Evaluation under Hidden Rule Changes in Minecraftcited by 1
  15. 2026UrbanGround: From Local Perception to Spatial Agency in a Real-Scale Citycited by 1
  16. 2026MineEvolve: Self-Evolution with Accumulated Knowledge for Long-Horizon Embodied Minecraft Agentscited by 0
  17. 2026World2Minecraft: Occupancy-Driven Simulated Scenes Constructioncited by 0
  18. 2026GROW: Aligning GRPO with State-Action Modeling for Open-World VLM Agentscited by 0
  19. 2026An Information-Theoretic Definition for Open-Ended Learningcited by 0
  20. 2026GameWAM: A World Action Model for Video Gamescited by 0
  21. 2026GameHorizon Suite: Multi-Horizon Data and Evaluation in Gameplaycited by 0