# The MineRL BASALT Competition on Learning from Human Feedback

arXiv: https://arxiv.org/abs/2107.01969 · HTML: https://mlsanity.com/paper/2107.01969/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 43 references that are in the map.
- First posted on arXiv 5 Jul 2021.

## Builds on (most cited first, 16 of 16)

- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Learning Robust Rewards with Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1710.11248/) (2017)
- [Large-Scale Study of Curiosity-Driven Learning](https://mlsanity.com/paper/1808.04355/) (2018)
- [Inverse Reward Design](https://mlsanity.com/paper/1711.02827/) (2017)
- [Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning](https://mlsanity.com/paper/1809.02925/) (2018)
- [Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spaces](https://mlsanity.com/paper/1709.10163/) (2017)
- [Reward-rational (implicit) choice: A unifying formalism for reward learning](https://mlsanity.com/paper/2002.04833/) (2020)
- [The MineRL 2020 Competition on Sample Efficient Reinforcement Learning using Human Priors](https://mlsanity.com/paper/2101.11071/) (2021)
- [Society-in-the-Loop: Programming the Algorithmic Social Contract](https://mlsanity.com/paper/1707.07232/) (2017)
- [Preferences Implicit in the State of the World](https://mlsanity.com/paper/1902.04198/) (2019)
- [CraftAssist: A Framework for Dialogue-enabled Interactive Agents](https://mlsanity.com/paper/1907.08584/) (2019)
- [Generative Design in Minecraft (GDMC), Settlement Generation Competition](https://mlsanity.com/paper/1803.09853/) (2018)
- [From Optimizing Engagement to Measuring Value](https://mlsanity.com/paper/2008.12623/) (2020)
- [The Multi-Agent Reinforcement Learning in Malm"O (MARL"O) Competition](https://mlsanity.com/paper/1901.08129/) (2019)

## Built on by (most cited first, 24 of 24)

- [MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge](https://mlsanity.com/paper/2206.08853/) (2022)
- [A Survey of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2312.14925/) (2023)
- [TEACh: Task-driven Embodied Agents that Chat](https://mlsanity.com/paper/2110.00534/) (2021)
- [Foundational Models Defining a New Era in Vision: A Survey and Outlook](https://mlsanity.com/paper/2307.13721/) (2023)
- [MCU: An Evaluation Framework for Open-Ended Game Agents](https://mlsanity.com/paper/2310.08367/) (2023)
- [Improving Grounded Language Understanding in a Collaborative Environment by Interacting with Agents Through Help Feedback](https://mlsanity.com/paper/2304.10750/) (2023)
- [A Survey of Explainable Reinforcement Learning](https://mlsanity.com/paper/2202.08434/) (2022)
- [MineRL Diamond 2021 Competition: Overview, Results, and Lessons Learned](https://mlsanity.com/paper/2202.10583/) (2022)
- [Towards Solving Fuzzy Tasks with Human Feedback: A Retrospective of the MineRL BASALT 2022 Competition](https://mlsanity.com/paper/2303.13512/) (2023)
- [Zero-shot Imitation Policy via Search in Demonstration Dataset](https://mlsanity.com/paper/2401.16398/) (2024)
- [Navigates Like Me: Understanding How People Evaluate Human-Like AI in Video Games](https://mlsanity.com/paper/2303.02160/) (2023)
- [PLAICraft: Large-Scale Time-Aligned Vision-Speech-Action Dataset for Embodied AI](https://mlsanity.com/paper/2505.12707/) (2025)
- [Behavioral Cloning via Search in Video PreTraining Latent Space](https://mlsanity.com/paper/2212.13326/) (2022)
- [BEDD: The MineRL BASALT Evaluation and Demonstrations Dataset for Training and Benchmarking Agents that Solve Fuzzy Tasks](https://mlsanity.com/paper/2312.02405/) (2023)
- [Retrospective on the 2021 BASALT Competition on Learning from Human Feedback](https://mlsanity.com/paper/2204.07123/) (2022)
- [Combining Learning from Human Feedback and Knowledge Engineering to Solve Hierarchical Tasks in Minecraft](https://mlsanity.com/paper/2112.03482/) (2021)
- [Behavioral Cloning via Search in Embedded Demonstration Dataset](https://mlsanity.com/paper/2306.09082/) (2023)
- [DIP-RL: Demonstration-Inferred Preference Learning in Minecraft](https://mlsanity.com/paper/2307.12158/) (2023)
- [A Survey on Complex Tasks for Goal-Directed Interactive Agents](https://mlsanity.com/paper/2409.18538/) (2024)
- [Improving Behavioural Cloning with Human-Driven Dynamic Dataset Augmentation](https://mlsanity.com/paper/2201.07719/) (2022)
- [Many Episode Learning in a Modular Embodied Agent via End-to-End Interaction](https://mlsanity.com/paper/2204.08687/) (2022)
- [Interaction is all You Need? A Study of Robots Ability to Understand and Execute](https://mlsanity.com/paper/2311.07150/) (2023)
- [Neural Abstructions: Abstractions that Support Construction for Grounded Language Learning](https://mlsanity.com/paper/2107.09285/) (2021)
- [PROWL: Prioritized Regret-Driven Optimization for World Model Learning](https://mlsanity.com/paper/2605.18803/) (2026)

## Abstract (arXiv)

The last decade has seen a significant increase of interest in deep learning research, with many public successes that have demonstrated its potential. As such, these systems are now being incorporated into commercial products. With this comes an additional challenge: how can we build AI systems that solve tasks where there is not a crisp, well-defined specification? While multiple solutions have been proposed, in this competition we focus on one in particular: learning from human feedback. Rather than training AI systems using a predefined reward function or using a labeled dataset with a predefined set of categories, we instead train the AI system using a learning signal derived from some form of human feedback, which can evolve over time as the understanding of the task changes, or as the capabilities of the AI system improve. The MineRL BASALT competition aims to spur forward research on this important class of techniques. We design a suite of four tasks in Minecraft for which we expect it will be hard to write down hardcoded reward functions. These tasks are defined by a paragraph of natural language: for example, "create a waterfall and take a scenic picture of it", with additional clarifying details. Participants must train a separate agent for each task, using any method they want. Agents are then evaluated by humans who have read the task description. To help participants get started, we provide a dataset of human demonstrations on each of the four tasks, as well as an imitation learning baseline that leverages these demonstrations. Our hope is that this competition will improve our ability to build AI systems that do what their designers intend them to do, even when the intent cannot be easily formalized. Besides allowing AI to solve more tasks, this can also enable more effective regulation of AI systems, as well as making progress on the value alignment problem.