# JueWu-MC: Playing Minecraft with Sample-efficient Hierarchical Reinforcement Learning

arXiv: https://arxiv.org/abs/2112.04907 · HTML: https://mlsanity.com/paper/2112.04907/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 32 of its 43 references that are in the map.
- First posted on arXiv 7 Dec 2021.

## Placed in

- Offline RL and imitation learning from data / [Imitation learning methods](https://mlsanity.com/offlinerl/imitation-learning-methods/)

## Builds on (most cited first, 30 of 32)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Exploration by Random Network Distillation](https://mlsanity.com/paper/1810.12894/) (2018)
- [Rainbow: Combining Improvements in Deep Reinforcement Learning](https://mlsanity.com/paper/1710.02298/) (2017)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [CURL: Contrastive Unsupervised Representations for Reinforcement Learning](https://mlsanity.com/paper/2004.04136/) (2020)
- [Overcoming Exploration in Reinforcement Learning with Demonstrations](https://mlsanity.com/paper/1709.10089/) (2017)
- [Google Research Football: A Novel Reinforcement Learning Environment](https://mlsanity.com/paper/1907.11180/) (2019)
- [Accelerating Reinforcement Learning with Learned Skill Priors](https://mlsanity.com/paper/2010.11944/) (2020)
- [MineRL: A Large-Scale Dataset of Minecraft Demonstrations](https://mlsanity.com/paper/1907.13440/) (2019)
- [Mastering Complex Control in MOBA Games with Deep Reinforcement Learning](https://mlsanity.com/paper/1912.09729/) (2019)
- [Self-Imitation Learning](https://mlsanity.com/paper/1806.05635/) (2018)
- [State Representation Learning for Control: An Overview](https://mlsanity.com/paper/1802.04181/) (2018)
- [Imitation Learning from Imperfect Demonstration](https://mlsanity.com/paper/1901.09387/) (2019)
- [Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning](https://mlsanity.com/paper/1706.05064/) (2017)
- [A Deep Hierarchical Approach to Lifelong Learning in Minecraft](https://mlsanity.com/paper/1604.07255/) (2016)
- [Hierarchical Imitation and Reinforcement Learning](https://mlsanity.com/paper/1803.00590/) (2018)
- [Control of Memory, Active Perception, and Action in Minecraft](https://mlsanity.com/paper/1605.09128/) (2016)
- [The MineRL 2020 Competition on Sample Efficient Reinforcement Learning using Human Priors](https://mlsanity.com/paper/2101.11071/) (2021)
- [Dynamics-aware Embeddings](https://mlsanity.com/paper/1908.09357/) (2019)
- [SEIHAI: A Sample-efficient Hierarchical AI for the MineRL Competition](https://mlsanity.com/paper/2111.08857/) (2021)
- [Supervised Learning Achieves Human-Level Performance in MOBA Games: A Case Study of Honor of Kings](https://mlsanity.com/paper/2011.12582/) (2020)
- [TiKick: Towards Playing Multi-agent Football Full Games from Single-agent Demonstrations](https://mlsanity.com/paper/2110.04507/) (2021)
- [Sample Efficient Reinforcement Learning through Learning from Demonstrations in Minecraft](https://mlsanity.com/paper/2003.06066/) (2020)
- [Scaling Imitation Learning in Minecraft](https://mlsanity.com/paper/2007.02701/) (2020)

## Built on by (most cited first, 28 of 28)

- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents](https://mlsanity.com/paper/2302.01560/) (2023)
- [Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos](https://mlsanity.com/paper/2206.11795/) (2022)
- [Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memory](https://mlsanity.com/paper/2305.17144/) (2023)
- [JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Models](https://mlsanity.com/paper/2311.05997/) (2023)
- [Language Models Meet World Models: Embodied Experiences Enhance Language Models](https://mlsanity.com/paper/2305.10626/) (2023)
- [MP5: A Multi-modal Open-ended Embodied System in Minecraft via Active Perception](https://mlsanity.com/paper/2312.07472/) (2023)
- [Auto MC-Reward: Automated Dense Reward Design with Large Language Models for Minecraft](https://mlsanity.com/paper/2312.09238/) (2023)
- [See and Think: Embodied Agent in Virtual Environment](https://mlsanity.com/paper/2311.15209/) (2023)
- [GROOT: Learning to Follow Instructions by Watching Gameplay Videos](https://mlsanity.com/paper/2310.08235/) (2023)
- [Open-World Multi-Task Control Through Goal-Aware Representation Learning and Adaptive Horizon Prediction](https://mlsanity.com/paper/2301.10034/) (2023)
- [TPTU-v2: Boosting Task Planning and Tool Usage of Large Language Model-based Agents in Real-world Systems](https://mlsanity.com/paper/2311.11315/) (2023)
- [MineRL Diamond 2021 Competition: Overview, Results, and Lessons Learned](https://mlsanity.com/paper/2202.10583/) (2022)
- [Odyssey: Empowering Minecraft Agents with Open-World Skills](https://mlsanity.com/paper/2407.15325/) (2024)
- [Hierarchical Auto-Organizing System for Open-Ended Multi-Agent Navigation](https://mlsanity.com/paper/2403.08282/) (2024)
- [S-Agents: Self-organizing Agents in Open-ended Environments](https://mlsanity.com/paper/2402.04578/) (2024)
- [Dynamics-Adaptive Continual Reinforcement Learning via Progressive Contextualization](https://mlsanity.com/paper/2209.00347/) (2022)
- [Reinforcement Learning Friendly Vision-Language Model for Minecraft](https://mlsanity.com/paper/2303.10571/) (2023)
- [Position: Interactive Generative Video as Next-Generation Game Engine](https://mlsanity.com/paper/2503.17359/) (2025)
- [UI-Voyager: A Self-Evolving GUI Agent Learning via Failed Experience](https://mlsanity.com/paper/2603.24533/) (2026)
- [Affordable Generative Agents](https://mlsanity.com/paper/2402.02053/) (2024)
- [MrSteve: Instruction-Following Agents in Minecraft with What-Where-When Memory](https://mlsanity.com/paper/2411.06736/) (2024)
- [RLogist: Fast Observation Strategy on Whole-slide Images with Deep Reinforcement Learning](https://mlsanity.com/paper/2212.01737/) (2022)
- [EscapeCraft: A 3D Room Escape Environment for Benchmarking Complex Multimodal Reasoning Ability](https://mlsanity.com/paper/2503.10042/) (2025)
- [Sample Dropout: A Simple yet Effective Variance Reduction Technique in Deep Policy Optimization](https://mlsanity.com/paper/2302.02299/) (2023)
- [Select before Act: Spatially Decoupled Action Repetition for Continuous Control](https://mlsanity.com/paper/2502.06919/) (2025)
- [Goal-Guided Efficient Exploration via Large Language Model in Reinforcement Learning](https://mlsanity.com/paper/2509.22008/) (2025)
- [Advantage-based Temporal Attack in Reinforcement Learning](https://mlsanity.com/paper/2602.19582/) (2026)

## Abstract (arXiv)

Learning rational behaviors in open-world games like Minecraft remains to be challenging for Reinforcement Learning (RL) research due to the compound challenge of partial observability, high-dimensional visual perception and delayed reward. To address this, we propose JueWu-MC, a sample-efficient hierarchical RL approach equipped with representation learning and imitation learning to deal with perception and exploration. Specifically, our approach includes two levels of hierarchy, where the high-level controller learns a policy to control over options and the low-level workers learn to solve each sub-task. To boost the learning of sub-tasks, we propose a combination of techniques including 1) action-aware representation learning which captures underlying relations between action and representation, 2) discriminator-based self-imitation learning for efficient exploration, and 3) ensemble behavior cloning with consistency filtering for policy robustness. Extensive experiments show that JueWu-MC significantly improves sample efficiency and outperforms a set of baselines by a large margin. Notably, we won the championship of the NeurIPS MineRL 2021 research competition and achieved the highest performance score ever.