# Nebula: A discourse aware Minecraft Builder

arXiv: https://arxiv.org/abs/2406.18164 · HTML: https://mlsanity.com/paper/2406.18164/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 21 references that are in the map.
- First posted on arXiv 26 Jun 2024.

## Builds on (most cited first, 9 of 9)

- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [Code as Policies: Language Model Programs for Embodied Control](https://mlsanity.com/paper/2209.07753/) (2022)
- [ProgPrompt: Generating Situated Robot Task Plans using Large Language Models](https://mlsanity.com/paper/2209.11302/) (2022)
- [Language to Rewards for Robotic Skill Synthesis](https://mlsanity.com/paper/2306.08647/) (2023)
- [Learning to Execute Actions or Ask Clarification Questions](https://mlsanity.com/paper/2204.08373/) (2022)
- [Interactive Grounded Language Understanding in a Collaborative Environment: IGLU 2021](https://mlsanity.com/paper/2205.02388/) (2022)
- [Transforming Human-Centered AI Collaboration: Redefining Embodied Agents Capabilities through Interactive Grounded Language Instructions](https://mlsanity.com/paper/2305.10783/) (2023)
- [Large Language Models as Minecraft Agents](https://mlsanity.com/paper/2402.08392/) (2024)

## Built on by (most cited first, 3 of 3)

- [LLMs Get Lost In Multi-Turn Conversation](https://mlsanity.com/paper/2505.06120/) (2025)
- [GuessBench: Sensemaking Multimodal Creativity in the Wild](https://mlsanity.com/paper/2506.00814/) (2025)
- [MineCEraft: Evaluating Language Models as Construction Engineers in the World of Minecraft](https://mlsanity.com/paper/2608.28884/) (2026)

## Abstract (arXiv)

When engaging in collaborative tasks, humans efficiently exploit the semantic structure of a conversation to optimize verbal and nonverbal interactions. But in recent "language to code" or "language to action" models, this information is lacking. We show how incorporating the prior discourse and nonlinguistic context of a conversation situated in a nonlinguistic environment can improve the "language to action" component of such interactions. We finetune an LLM to predict actions based on prior context; our model, Nebula, doubles the net-action F1 score over the baseline on this task of Jayannavar et al.(2020). We also investigate our model's ability to construct shapes and understand location descriptions using a synthetic dataset