# Retrospective on the 2021 BASALT Competition on Learning from Human Feedback

arXiv: https://arxiv.org/abs/2204.07123 · HTML: https://mlsanity.com/paper/2204.07123/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 14 of its 27 references that are in the map.
- First posted on arXiv 14 Apr 2022.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Reward modeling and RLHF](https://mlsanity.com/alignment/reward-modeling-and-rlhf/)

## Builds on (most cited first, 14 of 14)

- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Reward learning from human preferences and demonstrations in Atari](https://mlsanity.com/paper/1811.06521/) (2018)
- [Inverse Reward Design](https://mlsanity.com/paper/1711.02827/) (2017)
- [IQ-Learn: Inverse soft-Q Learning for Imitation](https://mlsanity.com/paper/2106.12142/) (2021)
- [Reward-rational (implicit) choice: A unifying formalism for reward learning](https://mlsanity.com/paper/2002.04833/) (2020)
- [Preferences Implicit in the State of the World](https://mlsanity.com/paper/1902.04198/) (2019)
- [CraftAssist: A Framework for Dialogue-enabled Interactive Agents](https://mlsanity.com/paper/1907.08584/) (2019)
- [The MineRL 2019 Competition on Sample Efficient Reinforcement Learning using Human Priors](https://mlsanity.com/paper/1904.10079/) (2019)
- [The MineRL BASALT Competition on Learning from Human Feedback](https://mlsanity.com/paper/2107.01969/) (2021)
- [Generative Design in Minecraft (GDMC), Settlement Generation Competition](https://mlsanity.com/paper/1803.09853/) (2018)
- [The Multi-Agent Reinforcement Learning in Malm"O (MARL"O) Competition](https://mlsanity.com/paper/1901.08129/) (2019)
- [Combining Learning from Human Feedback and Knowledge Engineering to Solve Hierarchical Tasks in Minecraft](https://mlsanity.com/paper/2112.03482/) (2021)

## Built on by (most cited first, 5 of 5)

- [Towards Solving Fuzzy Tasks with Human Feedback: A Retrospective of the MineRL BASALT 2022 Competition](https://mlsanity.com/paper/2303.13512/) (2023)
- [Reinforcement Learning Friendly Vision-Language Model for Minecraft](https://mlsanity.com/paper/2303.10571/) (2023)
- [BEDD: The MineRL BASALT Evaluation and Demonstrations Dataset for Training and Benchmarking Agents that Solve Fuzzy Tasks](https://mlsanity.com/paper/2312.02405/) (2023)
- [DIP-RL: Demonstration-Inferred Preference Learning in Minecraft](https://mlsanity.com/paper/2307.12158/) (2023)
- [Benchmarking Robustness and Generalization in Multi-Agent Systems: A Case Study on Neural MMO](https://mlsanity.com/paper/2308.15802/) (2023)

## Abstract (arXiv)

We held the first-ever MineRL Benchmark for Agents that Solve Almost-Lifelike Tasks (MineRL BASALT) Competition at the Thirty-fifth Conference on Neural Information Processing Systems (NeurIPS 2021). The goal of the competition was to promote research towards agents that use learning from human feedback (LfHF) techniques to solve open-world tasks. Rather than mandating the use of LfHF techniques, we described four tasks in natural language to be accomplished in the video game Minecraft, and allowed participants to use any approach they wanted to build agents that could accomplish the tasks. Teams developed a diverse range of LfHF algorithms across a variety of possible human feedback types. The three winning teams implemented significantly different approaches while achieving similar performance. Interestingly, their approaches performed well on different tasks, validating our choice of tasks to include in the competition. While the outcomes validated the design of our competition, we did not get as many participants and submissions as our sister competition, MineRL Diamond. We speculate about the causes of this problem and suggest improvements for future iterations of the competition.