mlsanityData to 7 Oct 2026

BEDD: The MineRL BASALT Evaluation and Demonstrations Dataset for Training and Benchmarking Agents that Solve Fuzzy Tasks

arXiv 2312.02405 ↗First posted 5 Dec 2023cs.AIcited by 8

Not on the map. Why?

Abstract (arXiv)

The MineRL BASALT competition has served to catalyze advances in learning from human feedback through four hard-to-specify tasks in Minecraft, such as create and photograph a waterfall. Given the completion of two years of BASALT competitions, we offer to the community a formalized benchmark through the BASALT Evaluation and Demonstrations Dataset (BEDD), which serves as a resource for algorithm development and performance assessment. BEDD consists of a collection of 26 million image-action pairs from nearly 14,000 videos of human players completing the BASALT tasks in Minecraft. It also includes over 3,000 dense pairwise human evaluations of human and algorithmic agents. These comparisons serve as a fixed, preliminary leaderboard for evaluating newly-developed algorithms. To enable this comparison, we present a streamlined codebase for benchmarking new algorithms against the leaderboard. In addition to presenting these datasets, we conduct a detailed analysis of the data from both datasets to guide algorithm development and evaluation. The released code and data are available at https://github.com/minerllabs/basalt-benchmark .

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 37 of its 46 references that are in the map. First posted on arXiv 5 Dec 2023.

Builds on

37 of 46 references.

References · 37

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  3. 2017Deep reinforcement learning from human preferencescited by 3,846
  4. 2020Learning to summarize from human feedbackcited by 2,562
  5. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  6. 2018Datasheets for Datasetscited by 1,371
  7. 2023Mastering Diverse Domains through World Modelscited by 1,193
  8. 2018DeepMind Control Suitecited by 935
  9. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
  10. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  11. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
  12. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
25 more
  1. 2018Reward learning from human preferences and demonstrations in Ataricited by 314
  2. 2023Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memorycited by 288
  3. 2018Large-Scale Study of Curiosity-Driven Learningcited by 277
  4. 2021Benchmarking the Spectrum of Agent Capabilitiescited by 193
  5. 2020Reward-rational (implicit) choice: A unifying formalism for reward learningcited by 119
  6. 2021ThriftyDAgger: Budget-Aware Novelty and Risk Gating for Interactive Imitation Learningcited by 98
  7. 2023STEVE-1: A Generative Model for Text-to-Behavior in Minecraftcited by 95
  8. 2019Deep Reinforcement Learning from Policy-Dependent Human Feedbackcited by 56
  9. 2023Empirical Design in Reinforcement Learningcited by 54
  10. 2020Action Space Shaping in Deep Reinforcement Learningcited by 45
  11. 2022imitation: Clean Imitation Learning Implementationscited by 34
  12. 2019CraftAssist: A Framework for Dialogue-enabled Interactive Agentscited by 27
  13. 2019Risk-Aware Active Inverse Reinforcement Learningcited by 26
  14. 2019The MineRL 2019 Competition on Sample Efficient Reinforcement Learning using Human Priorscited by 25
  15. 2021The MineRL BASALT Competition on Learning from Human Feedbackcited by 24
  16. 2020EvoCraft: A New Challenge for Open-Endednesscited by 23
  17. 2018Generative Design in Minecraft (GDMC), Settlement Generation Competitioncited by 22
  18. 2022Use-Case-Grounded Simulations for Explanation Evaluationcited by 19
  19. 2021Navigation Turing Test (NTT): Learning to Evaluate Human-Like Navigationcited by 16
  20. 2021NeurIPS 2021 Competition IGLU: Interactive Grounded Language Understanding in a Collaborative Environmentcited by 15
  21. 2023Towards Solving Fuzzy Tasks with Human Feedback: A Retrospective of the MineRL BASALT 2022 Competitioncited by 15
  22. 2019The Multi-Agent Reinforcement Learning in Malm"O (MARL"O) Competitioncited by 14
  23. 2022IGLU 2022: Interactive Grounded Language Understanding in a Collaborative Environment at NeurIPS 2022cited by 14
  24. 2022Behavioral Cloning via Search in Video PreTraining Latent Spacecited by 9
  25. 2022Retrospective on the 2021 BASALT Competition on Learning from Human Feedbackcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 1; 2026: 2.

8 papers.

  1. 2025PLAICraft: Large-Scale Time-Aligned Vision-Speech-Action Dataset for Embodied AIcited by 10
  2. 2024Online Adaptation for Enhancing Imitation Learning Policiescited by 3
  3. 2024Beyond the Sum: Unlocking AI Agents Potential Through Market Forcescited by 3
  4. 2026MineExplorer: Evaluating Open-World Exploration of MLLM Agents in Minecraftcited by 3
  5. 2024A Survey on Complex Tasks for Goal-Directed Interactive Agentscited by 2
  6. 2024IDAT: A Multi-Modal Dataset and Toolkit for Building and Evaluating Interactive Task-Solving Agentscited by 1
  7. 2024ViSTa Dataset: Do vision-language models understand sequential tasks?cited by 0
  8. 2026PROWL: Prioritized Regret-Driven Optimization for World Model Learningcited by 0