# Towards Solving Fuzzy Tasks with Human Feedback: A Retrospective of the MineRL BASALT 2022 Competition

arXiv: https://arxiv.org/abs/2303.13512 · HTML: https://mlsanity.com/paper/2303.13512/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 25 of its 34 references that are in the map.
- First posted on arXiv 23 Mar 2023.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Reward modeling and RLHF](https://mlsanity.com/alignment/reward-modeling-and-rlhf/)

## Builds on (most cited first, 25 of 25)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge](https://mlsanity.com/paper/2206.08853/) (2022)
- [The Option-Critic Architecture](https://mlsanity.com/paper/1609.05140/) (2016)
- [Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos](https://mlsanity.com/paper/2206.11795/) (2022)
- [Benchmarking the Spectrum of Agent Capabilities](https://mlsanity.com/paper/2109.06780/) (2021)
- [IQ-Learn: Inverse soft-Q Learning for Imitation](https://mlsanity.com/paper/2106.12142/) (2021)
- [imitation: Clean Imitation Learning Implementations](https://mlsanity.com/paper/2211.11972/) (2022)
- [CraftAssist: A Framework for Dialogue-enabled Interactive Agents](https://mlsanity.com/paper/1907.08584/) (2019)
- [The MineRL 2019 Competition on Sample Efficient Reinforcement Learning using Human Priors](https://mlsanity.com/paper/1904.10079/) (2019)
- [The MineRL BASALT Competition on Learning from Human Feedback](https://mlsanity.com/paper/2107.01969/) (2021)
- [UniMASK: Unified Inference in Sequential Decision Problems](https://mlsanity.com/paper/2211.10869/) (2022)
- [EvoCraft: A New Challenge for Open-Endedness](https://mlsanity.com/paper/2012.04751/) (2020)
- [Generative Design in Minecraft (GDMC), Settlement Generation Competition](https://mlsanity.com/paper/1803.09853/) (2018)
- [NeurIPS 2021 Competition IGLU: Interactive Grounded Language Understanding in a Collaborative Environment](https://mlsanity.com/paper/2110.06536/) (2021)
- [The Multi-Agent Reinforcement Learning in Malm"O (MARL"O) Competition](https://mlsanity.com/paper/1901.08129/) (2019)
- [Behavioral Cloning via Search in Video PreTraining Latent Space](https://mlsanity.com/paper/2212.13326/) (2022)
- [Retrospective on the 2021 BASALT Competition on Learning from Human Feedback](https://mlsanity.com/paper/2204.07123/) (2022)

## Built on by (most cited first, 15 of 15)

- [Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memory](https://mlsanity.com/paper/2305.17144/) (2023)
- [Large Language Models and Games: A Survey and Roadmap](https://mlsanity.com/paper/2402.18659/) (2024)
- [Creative Agents: Empowering Agents with Imagination for Creative Tasks](https://mlsanity.com/paper/2312.02519/) (2023)
- [Zero-shot Imitation Policy via Search in Demonstration Dataset](https://mlsanity.com/paper/2401.16398/) (2024)
- [AssistanceZero: Scalably Solving Assistance Games](https://mlsanity.com/paper/2504.07091/) (2025)
- [BEDD: The MineRL BASALT Evaluation and Demonstrations Dataset for Training and Benchmarking Agents that Solve Fuzzy Tasks](https://mlsanity.com/paper/2312.02405/) (2023)
- [Benchmarks for Physical Reasoning AI](https://mlsanity.com/paper/2312.10728/) (2023)
- [Natural Language as Policies: Reasoning for Coordinate-Level Embodied Control with LLMs](https://mlsanity.com/paper/2403.13801/) (2024)
- [Interpretability in Action: Exploratory Analysis of VPT, a Minecraft Agent](https://mlsanity.com/paper/2407.12161/) (2024)
- [Behavioral Cloning via Search in Embedded Demonstration Dataset](https://mlsanity.com/paper/2306.09082/) (2023)
- [DIP-RL: Demonstration-Inferred Preference Learning in Minecraft](https://mlsanity.com/paper/2307.12158/) (2023)
- [Aligning Agents like Large Language Models](https://mlsanity.com/paper/2406.04208/) (2024)
- [SOAP-RL: Sequential Option Advantage Propagation for Reinforcement Learning in POMDP Environments](https://mlsanity.com/paper/2407.18913/) (2024)
- [Object and Contact Point Tracking in Demonstrations Using 3D Gaussian Splatting](https://mlsanity.com/paper/2411.03555/) (2024)
- [STEVE-Audio: Expanding the Goal Conditioning Modalities of Embodied Agents in Minecraft](https://mlsanity.com/paper/2412.00949/) (2024)

## Abstract (arXiv)

To facilitate research in the direction of fine-tuning foundation models from human feedback, we held the MineRL BASALT Competition on Fine-Tuning from Human Feedback at NeurIPS 2022. The BASALT challenge asks teams to compete to develop algorithms to solve tasks with hard-to-specify reward functions in Minecraft. Through this competition, we aimed to promote the development of algorithms that use human feedback as channels to learn the desired behavior. We describe the competition and provide an overview of the top solutions. We conclude by discussing the impact of the competition and future directions for improvement.