# MineCEraft: Evaluating Language Models as Construction Engineers in the World of Minecraft

arXiv: https://arxiv.org/abs/2608.28884 · HTML: https://mlsanity.com/paper/2608.28884/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 29 references that are in the map.
- First posted on arXiv 28 Aug 2026.

## Builds on (most cited first, 14 of 14)

- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks](https://mlsanity.com/paper/1912.01734/) (2019)
- [Vision-and-Dialog Navigation](https://mlsanity.com/paper/1907.04957/) (2019)
- [EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents](https://mlsanity.com/paper/2502.09560/) (2025)
- [TEACh: Task-driven Embodied Agents that Chat](https://mlsanity.com/paper/2110.00534/) (2021)
- [Gold-medalist Performance in Solving Olympiad Geometry with AlphaGeometry2](https://mlsanity.com/paper/2502.03544/) (2025)
- [FrontierScience: Evaluating AI's Ability to Perform Expert-Level Scientific Tasks](https://mlsanity.com/paper/2601.21165/) (2026)
- [CraftAssist: A Framework for Dialogue-enabled Interactive Agents](https://mlsanity.com/paper/1907.08584/) (2019)
- [Collaborating Action by Action: A Multi-agent LLM Framework for Embodied Reasoning](https://mlsanity.com/paper/2504.17950/) (2025)
- [CEQuest: Benchmarking Large Language Models for Construction Estimation](https://mlsanity.com/paper/2508.16081/) (2025)
- [Retrieval-Augmented Code Generation for Situated Action Generation: A Case Study on Minecraft](https://mlsanity.com/paper/2406.17553/) (2024)
- [Nebula: A discourse aware Minecraft Builder](https://mlsanity.com/paper/2406.18164/) (2024)
- [BAP v2: An Enhanced Task Framework for Instruction Following in Minecraft Dialogues](https://mlsanity.com/paper/2501.10836/) (2025)
- [Elasticity Solver in Minecraft for Learning Mechanics of Materials by Gaming](https://mlsanity.com/paper/2212.08124/) (2022)

## Abstract (arXiv)

We introduce MineCEraft (Minecraft Construction Engineering Benchmark, pronounced mine-see-ee-raft), an easy-to-use, open-source benchmark designed to systematically evaluate the reliability and limitations of LLMs for construction tasks in Minecraft. The MineCEraft benchmark comprises 723 domain-expert hand-crafted natural-language instructions with programmatically verifiable evaluation, spanning 17 distinct task categories, providing a safe and controllable experimental environment for assessing LLMs' ability to perform realistic construction engineering tasks. With this benchmark, we conduct an in-depth evaluation of state-of-the-art LLMs and perform a detailed error analysis, revealing key failure modes and practical challenges in applying LLMs to construction engineering tasks.