# A LLM Benchmark based on the Minecraft Builder Dialog Agent Task

arXiv: https://arxiv.org/abs/2407.12734 · HTML: https://mlsanity.com/paper/2407.12734/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use.
- Builds on 14 of its 21 references that are in the map.
- First posted on arXiv 17 Jul 2024.

## Placed in

- LLM agents and tool use / [Reinforcement learning for LLM agents](https://mlsanity.com/agents/reinforcement-learning-for-llm-agents/)

## Builds on (most cited first, 14 of 14)

- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks](https://mlsanity.com/paper/1502.05698/) (2015)
- [SmartPlay: A Benchmark for LLMs as Intelligent Agents](https://mlsanity.com/paper/2310.01557/) (2023)
- [Learning to Speak and Act in a Fantasy Text Adventure Game](https://mlsanity.com/paper/1903.03094/) (2019)
- [StepGame: A New Benchmark for Robust Multi-Hop Spatial Reasoning in Texts](https://mlsanity.com/paper/2204.08292/) (2022)
- [Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence](https://mlsanity.com/paper/2210.07109/) (2022)
- [Transfer Learning with Synthetic Corpora for Spatial Role Labeling and Reasoning](https://mlsanity.com/paper/2210.16952/) (2022)
- [CraftAssist: A Framework for Dialogue-enabled Interactive Agents](https://mlsanity.com/paper/1907.08584/) (2019)
- [Interactive Grounded Language Understanding in a Collaborative Environment: IGLU 2021](https://mlsanity.com/paper/2205.02388/) (2022)
- [Why Build an Assistant in Minecraft?](https://mlsanity.com/paper/1907.09273/) (2019)
- [Collecting Interactive Multi-modal Datasets for Grounded Language Understanding](https://mlsanity.com/paper/2211.06552/) (2022)
- [Large Language Models as Minecraft Agents](https://mlsanity.com/paper/2402.08392/) (2024)
- [Retrieval-Augmented Code Generation for Situated Action Generation: A Case Study on Minecraft](https://mlsanity.com/paper/2406.17553/) (2024)

## Built on by (most cited first, 4 of 4)

- [GameArena: Evaluating LLM Reasoning through Live Computer Games](https://mlsanity.com/paper/2412.06394/) (2024)
- [MTR-Bench: A Comprehensive Benchmark for Multi-Turn Reasoning Evaluation](https://mlsanity.com/paper/2505.17123/) (2025)
- [Reflection-Bench: Evaluating Epistemic Agency in Large Language Models](https://mlsanity.com/paper/2410.16270/) (2024)
- [Referential ambiguity and clarification requests: comparing human and LLM behaviour](https://mlsanity.com/paper/2507.10445/) (2025)

## Abstract (arXiv)

In this work we proposing adapting the Minecraft builder task into an LLM benchmark suitable for evaluating LLM ability in spatially orientated tasks, and informing builder agent design. Previous works have proposed corpora with varying complex structures, and human written instructions. We instead attempt to provide a comprehensive synthetic benchmark for testing builder agents over a series of distinct tasks that comprise of common building operations. We believe this approach allows us to probe specific strengths and weaknesses of different agents, and test the ability of LLMs in the challenging area of spatial reasoning and vector based math.