# BEDD: The MineRL BASALT Evaluation and Demonstrations Dataset for Training and Benchmarking Agents that Solve Fuzzy Tasks

arXiv: https://arxiv.org/abs/2312.02405 · HTML: https://mlsanity.com/paper/2312.02405/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 46 references that are in the map.
- First posted on arXiv 5 Dec 2023.

## Builds on (most cited first, 30 of 37)

- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [Mastering Diverse Domains through World Models](https://mlsanity.com/paper/2301.04104/) (2023)
- [DeepMind Control Suite](https://mlsanity.com/paper/1801.00690/) (2018)
- [Leveraging Procedural Generation to Benchmark Reinforcement Learning](https://mlsanity.com/paper/1912.01588/) (2019)
- [MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge](https://mlsanity.com/paper/2206.08853/) (2022)
- [Benchmarking Deep Reinforcement Learning for Continuous Control](https://mlsanity.com/paper/1604.06778/) (2016)
- [Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos](https://mlsanity.com/paper/2206.11795/) (2022)
- [Reward learning from human preferences and demonstrations in Atari](https://mlsanity.com/paper/1811.06521/) (2018)
- [Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memory](https://mlsanity.com/paper/2305.17144/) (2023)
- [Large-Scale Study of Curiosity-Driven Learning](https://mlsanity.com/paper/1808.04355/) (2018)
- [Benchmarking the Spectrum of Agent Capabilities](https://mlsanity.com/paper/2109.06780/) (2021)
- [Reward-rational (implicit) choice: A unifying formalism for reward learning](https://mlsanity.com/paper/2002.04833/) (2020)
- [ThriftyDAgger: Budget-Aware Novelty and Risk Gating for Interactive Imitation Learning](https://mlsanity.com/paper/2109.08273/) (2021)
- [STEVE-1: A Generative Model for Text-to-Behavior in Minecraft](https://mlsanity.com/paper/2306.00937/) (2023)
- [Deep Reinforcement Learning from Policy-Dependent Human Feedback](https://mlsanity.com/paper/1902.04257/) (2019)
- [Empirical Design in Reinforcement Learning](https://mlsanity.com/paper/2304.01315/) (2023)
- [Action Space Shaping in Deep Reinforcement Learning](https://mlsanity.com/paper/2004.00980/) (2020)
- [imitation: Clean Imitation Learning Implementations](https://mlsanity.com/paper/2211.11972/) (2022)
- [CraftAssist: A Framework for Dialogue-enabled Interactive Agents](https://mlsanity.com/paper/1907.08584/) (2019)
- [Risk-Aware Active Inverse Reinforcement Learning](https://mlsanity.com/paper/1901.02161/) (2019)
- [The MineRL 2019 Competition on Sample Efficient Reinforcement Learning using Human Priors](https://mlsanity.com/paper/1904.10079/) (2019)
- [The MineRL BASALT Competition on Learning from Human Feedback](https://mlsanity.com/paper/2107.01969/) (2021)
- [EvoCraft: A New Challenge for Open-Endedness](https://mlsanity.com/paper/2012.04751/) (2020)
- [Generative Design in Minecraft (GDMC), Settlement Generation Competition](https://mlsanity.com/paper/1803.09853/) (2018)
- [Use-Case-Grounded Simulations for Explanation Evaluation](https://mlsanity.com/paper/2206.02256/) (2022)

## Built on by (most cited first, 8 of 8)

- [PLAICraft: Large-Scale Time-Aligned Vision-Speech-Action Dataset for Embodied AI](https://mlsanity.com/paper/2505.12707/) (2025)
- [Beyond the Sum: Unlocking AI Agents Potential Through Market Forces](https://mlsanity.com/paper/2501.10388/) (2024)
- [MineExplorer: Evaluating Open-World Exploration of MLLM Agents in Minecraft](https://mlsanity.com/paper/2605.30931/) (2026)
- [Online Adaptation for Enhancing Imitation Learning Policies](https://mlsanity.com/paper/2406.04913/) (2024)
- [A Survey on Complex Tasks for Goal-Directed Interactive Agents](https://mlsanity.com/paper/2409.18538/) (2024)
- [IDAT: A Multi-Modal Dataset and Toolkit for Building and Evaluating Interactive Task-Solving Agents](https://mlsanity.com/paper/2407.08898/) (2024)
- [ViSTa Dataset: Do vision-language models understand sequential tasks?](https://mlsanity.com/paper/2411.13211/) (2024)
- [PROWL: Prioritized Regret-Driven Optimization for World Model Learning](https://mlsanity.com/paper/2605.18803/) (2026)

## Abstract (arXiv)

The MineRL BASALT competition has served to catalyze advances in learning from human feedback through four hard-to-specify tasks in Minecraft, such as create and photograph a waterfall. Given the completion of two years of BASALT competitions, we offer to the community a formalized benchmark through the BASALT Evaluation and Demonstrations Dataset (BEDD), which serves as a resource for algorithm development and performance assessment. BEDD consists of a collection of 26 million image-action pairs from nearly 14,000 videos of human players completing the BASALT tasks in Minecraft. It also includes over 3,000 dense pairwise human evaluations of human and algorithmic agents. These comparisons serve as a fixed, preliminary leaderboard for evaluating newly-developed algorithms. To enable this comparison, we present a streamlined codebase for benchmarking new algorithms against the leaderboard. In addition to presenting these datasets, we conduct a detailed analysis of the data from both datasets to guide algorithm development and evaluation. The released code and data are available at https://github.com/minerllabs/basalt-benchmark .