mlsanityData to 7 Oct 2026

Combining Learning from Human Feedback and Knowledge Engineering to Solve Hierarchical Tasks in Minecraft

arXiv 2112.03482 ↗First posted 7 Dec 2021cs.LG cs.AI cs.HCcited by 3

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Real-world tasks of interest are generally poorly defined by human-readable descriptions and have no pre-defined reward signals unless it is defined by a human designer. Conversely, data-driven algorithms are often designed to solve a specific, narrowly defined, task with performance metrics that drives the agent's learning. In this work, we present the solution that won first place and was awarded the most human-like agent in the 2021 NeurIPS Competition MineRL BASALT Challenge: Learning from Human Feedback in Minecraft, which challenged participants to use human data to solve four tasks defined only by a natural language description and no reward function. Our approach uses the available human demonstration data to train an imitation learning policy for navigation and additional human feedback to train an image classifier. These modules, combined with an estimated odometry map, become a powerful state-machine designed to utilize human knowledge in a natural hierarchical paradigm. We compare this hybrid intelligence approach to both end-to-end machine learning and pure engineered solutions, which are then judged by human evaluators. Codebase is available at https://github.com/viniciusguigo/kairos_minerl_basalt.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 31 of its 52 references that are in the map. First posted on arXiv 7 Dec 2021.

Builds on

31 of 52 references.

Closest ancestors · 1

Same direction, newest first

  1. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 30

  1. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  4. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  5. 2016Generative Adversarial Imitation Learningcited by 1,576
  6. 2016End to End Learning for Self-Driving Carscited by 1,000
  7. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  8. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  9. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
  10. 2021Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learningcited by 395
  11. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 328
  12. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
18 more
  1. 2019MineRL: A Large-Scale Dataset of Minecraft Demonstrationscited by 210
  2. 2017Interactive Learning from Policy-Dependent Human Feedbackcited by 155
  3. 2017Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spacescited by 122
  4. 2017Trial without Error: Towards Safe Reinforcement Learning via Human Interventioncited by 102
  5. 2021GANcraft: Unsupervised 3D Neural Rendering of Minecraft Worldscited by 98
  6. 2017Combining Self-Supervised Learning and Imitation for Vision-Based Rope Manipulationcited by 90
  7. 2021Hybrid Intelligencecited by 55
  8. 2021The MineRL 2020 Competition on Sample Efficient Reinforcement Learning using Human Priorscited by 40
  9. 2021Replacing Rewards with Examples: Example-Based Policy Search via Recursive Classificationcited by 38
  10. 2021The MineRL BASALT Competition on Learning from Human Feedbackcited by 24
  11. 2020Inverse Reinforcement Learning with Natural Language Goalscited by 20
  12. 2019Integrating Behavior Cloning and Reinforcement Learning for Improved Performance in Dense and Sparse Reward Environmentscited by 17
  13. 2020Scaling Imitation Learning in Minecraftcited by 12
  14. 2018Efficiently Combining Human Demonstrations and Interventions for Safe Training of Autonomous Systems in Real-Timecited by 7
  15. 2019Learning Your Way Without Map or Compass: Panoramic Target Driven Visual Navigationcited by 7
  16. 2020Human-in-the-Loop Methods for Data-Driven and Reinforcement Learning Systemscited by 4
  17. 2016From virtual demonstration to real-world manipulation using LSTM and MDNcited by 3
  18. 2018Cycle-of-Learning for Autonomous Systems from Human Interactioncited by 3

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 2.

3 papers.

  1. 2023LLM Augmented Hierarchical Agentscited by 9
  2. 2022Retrospective on the 2021 BASALT Competition on Learning from Human Feedbackcited by 5
  3. 2023DIP-RL: Demonstration-Inferred Preference Learning in Minecraftcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2023Towards Solving Fuzzy Tasks with Human Feedback: A Retrospective of the MineRL BASALT 2022 Competition
  2. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge
  3. 2022Behavioral Cloning via Search in Video PreTraining Latent Space
  4. 2019Deep Reinforcement Learning from Policy-Dependent Human Feedback
  5. 2024Zero-shot Imitation Policy via Search in Demonstration Dataset
  6. 2020Playing Minecraft with Behavioural Cloning
  7. 2023Behavioral Cloning via Search in Embedded Demonstration Dataset
  8. 2025AssistanceZero: Scalably Solving Assistance Games