mlsanityData to 7 Oct 2026

MineAnyBuild: Benchmarking Spatial Planning for Open-world AI Agents

arXiv 2505.20148 ↗First posted 26 May 2025cs.AIcited by 3

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Spatial Planning is a crucial part in the field of spatial intelligence, which requires the understanding and planning about object arrangements in space perspective. AI agents with the spatial planning ability can better adapt to various real-world applications, including robotic manipulation, automatic assembly, urban planning etc. Recent works have attempted to construct benchmarks for evaluating the spatial intelligence of Multimodal Large Language Models (MLLMs). Nevertheless, these benchmarks primarily focus on spatial reasoning based on typical Visual Question-Answering (VQA) forms, which suffers from the gap between abstract spatial understanding and concrete task execution. In this work, we take a step further to build a comprehensive benchmark called MineAnyBuild, aiming to evaluate the spatial planning ability of open-world AI agents in the Minecraft game. Specifically, MineAnyBuild requires an agent to generate executable architecture building plans based on the given multi-modal human instructions. It involves 4,000 curated spatial planning tasks and also provides a paradigm for infinitely expandable data collection by utilizing rich player-generated content. MineAnyBuild evaluates spatial planning through four core supporting dimensions: spatial understanding, spatial reasoning, creativity, and spatial commonsense. Based on MineAnyBuild, we perform a comprehensive evaluation for existing MLLM-based agents, revealing the severe limitations but enormous potential in their spatial planning abilities. We believe our MineAnyBuild will open new avenues for the evaluation of spatial intelligence and help promote further development for open-world AI agents capable of spatial planning.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 42 of its 61 references that are in the map. First posted on arXiv 26 May 2025.

Builds on

42 of 61 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Optimus-2: Multimodal Minecraft Agent with Goal-Observation-Action Conditioned Policy27 Feb
  2. 2024APT: Architectural Planning and Text-to-Blueprint Construction Using Large Language Models for Open-World Agents26 Nov
  3. 2024Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Tasks7 Aug
  4. 2023JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Models10 Nov
  5. 2023Voyager: An Open-Ended Embodied Agent with Large Language Models25 May
  6. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge17 Jun

Other references · 36

  1. 2023Visual Instruction Tuningcited by 7,792
  2. 2025Qwen2.5-VL Technical Reportcited by 5,418
  3. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  4. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  5. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  6. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 679
  7. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 580
  8. 2025SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Modelcited by 502
  9. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 396
  10. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
  11. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Modelscited by 297
  12. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
24 more
  1. 2019MineRL: A Large-Scale Dataset of Minecraft Demonstrationscited by 210
  2. 2024VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Taskscited by 171
  3. 2024RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Roboticscited by 133
  4. 2024EmbSpatial-Bench: Benchmarking Spatial Understanding for Embodied Tasks with Large Vision-Language Modelscited by 100
  5. 2023STEVE-1: A Generative Model for Text-to-Behavior in Minecraftcited by 95
  6. 2025MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraftcited by 93
  7. 2025STI-Bench: Are MLLMs Ready for Precise Spatial-Temporal World Understanding?cited by 74
  8. 2023MP5: A Multi-modal Open-ended Embodied System in Minecraft via Active Perceptioncited by 67
  9. 2024SeeGround: See and Ground for Zero-Shot Open-Vocabulary 3D Visual Groundingcited by 66
  10. 2025SpatialCoT: Advancing Spatial Reasoning through Coordinate Alignment and Chain-of-Thought for Embodied Task Planningcited by 62
  11. 2025LEGO-Puzzles: How Good Are MLLMs at Multi-Step Spatial Reasoning?cited by 49
  12. 2022Things not Written in Text: Exploring Spatial Commonsense from Visual Signalscited by 41
  13. 2023Large Language Model Instruction Following: A Survey of Progresses and Challengescited by 31
  14. 2024SPARTUN3D: Situated Spatial Understanding of 3D World in Large Language Modelscited by 26
  15. 2024ROCKET-1: Mastering Open-World Interaction with Visual-Temporal Context Promptingcited by 20
  16. 2024TeamCraft: A Benchmark for Multi-Modal Multi-Agent Systems in Minecraftcited by 20
  17. 2024DreamCraft: Text-Guided Generation of Functional 3D Environments in Minecraftcited by 17
  18. 2024MineStudio: A Streamlined Package for Minecraft AI Agent Developmentcited by 12
  19. 2017Acquiring Common Sense Spatial Knowledge through Implicit Spatial Templatescited by 10
  20. 2022Open-Ended Evolution for Minecraft Building Generationcited by 9
  21. 2024GROOT-2: Weakly Supervised Multi-Modal Instruction Following Agentscited by 9
  22. 2025ROCKET-2: Steering Visuomotor Policy via Cross-View Goal Alignmentcited by 6
  23. 2015On the ranking of a Swiss system chess team tournamentcited by 4
  24. 20243D Building Generation in Minecraft via Large Language Modelscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026MineExplorer: Evaluating Open-World Exploration of MLLM Agents in Minecraftcited by 3
  2. 2026MirrorCraft: Paired Evaluation under Hidden Rule Changes in Minecraftcited by 1
  3. 2026Completion at the Boundary (CaB): Deployable Switching with Completion-Aware Control under Limited Calibrationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A LLM Benchmark based on the Minecraft Builder Dialog Agent Task
  2. 2025GVGAI-LLM: Evaluating Large Language Model Agents with Infinite Games
  3. 2026SpatialWorld: Benchmarking Interactive Spatial Reasoning of Multimodal Agents in Real-World Tasks
  4. 2025MARBLE: A Hard Benchmark for Multimodal Spatial Reasoning and Planning
  5. 2024GRASP: A Grid-Based Benchmark for Evaluating Commonsense Spatial Reasoning
  6. 2025SC2Arena and StarEvolve: Benchmark and Self-Improvement Framework for LLMs in Complex Decision-Making Tasks
  7. 2026Spatial Reasoning in LLM Game Agents: Impact of Causal Context and Multi-Step Planning
  8. 2025CubeBench: Diagnosing Interactive, Long-Horizon Spatial Reasoning Under Partial Observations